Terraining is quite the art. Check out Mipsy's
Ellirian Isles project and his description of his procedures at this thread. I'll post his details right here, though I'm not sure how clear they'll be (I find them kind of hard to follow myself, but they'll at least give you a general idea of how to terrain.)
Terrain Layer Settings (in this order)
LeavesOakGreen_A
Roots_Green_A
RoadStones_C
RoadStones_Mossy_A
FlowersWhite_A
FlowersYellow_A
FoliageGreen_A
GrassGreen_D
GrassGreen_A
CoastalStones_C
CoastalStones_D
CoastalStones_A
CoastalStones_B
SandBrown_A
SandBrown_B
SandWhite_A
RockMountain_B
MudLightBrown_A
Beach Settings
1.0 White Sand A for Base
CTRL + 0.40 Brown Sand B near Shore
Place Rocks
CTRL + 0.80 Coastal Rock C near Rocks
CTRL + 0.40 Coastal Rock D near Rocks
CTRL + 0.40 Coastal Rock A near Rocks
CTRL + 0.40 Brown Sand A near Rocks towards Shore
CTRL + 0.70 Coastal Rock B near Shore towards Rocks
Clean Brown Sand A and B
Place Seaweed
Place Dune Grass
Place Coral
Place Shells
Place Starfish
Place Seagulls FX
Coastal Forest Settings
1.0 Light Brown Mud for Base
1.0 Rock Mountain for Borders
CTRL + 0.25 Grass A
CTRL + 0.25 Grass D
Place Trees
CTRL + 0.25 Green Root
CTRL + 0.20 Grass A Borders
CTRL + 0.20 Green Root Borders on cliffs near Trees
CTRL + 0.25 Oak Green Leaves
CTRL + 0.50 White/Yellow Flowers
Place Rocks
Place Bushes
Place Mushrooms and Flowers
CTRL + 0.20 Coastal Rock A Path
CTRL + 0.20 Grass A Clean
Add Cliffs
Add Canopy Leaves
Beach to Coastal Forest Transition Settings
1.0 Light Brown Mud base
CTRL + 0.80 Coastal Stones C at bottom of ramp
CTRL + 0.25 to 0.20 Grass A receding from upper to middle of ramp
CTRL + 0.20 Grass D receding from upper to middle of ramp
CTRL + 0.40 Brown Sand A receding from bottom to middle of ramp
CTRL + 0.40 Coastal Stones C&D receding from bottom to middle of ramp
CTRL + 0.40 Coastal Stones B receding from sides of ramp to middle.
The numbers refer to clamp settings.
Cleaning means erasing with 1 strength.
Terraining takes a long time to get the hang of. The basic idea is to shape your terrain first and then apply textures. Build your mountains, create your valleys, and ensure there's plenty of places to walk around and it's not just a little corridor for the player to walk around in. For shaping, take advantage of smoothing (shift) to eliminate harsh edges. Remember that a lot of the mountain feel comes from placing rocks and whatnot.
For best results, but more work, you'll want to make the map big enough so the player can never see the edge of the map. You'll have to place CameraLock triggers to prevent the player from going beyond the map's bounds. You'll have to have some borders to prevent the player from walking beyond the CameraLock triggers, and try to make these as natural as possible instead of having tons of invisible walls (by painting the AI grid so that the player can't walk there even though they think they should be able to.)
I recommend making a large terrain (at least 256 by 256) to begin with even if you think your zone won't be that big. You'd rather have the space to expand than have some unused space that the player won't end up seeing anyways. It's possible to "attach" terrains to another terrain so the player can walk from one to another, but this is extremely tedious and should be avoided at all costs. It's much preferable to just make a zone, and then if you run out of space, make a teleport trigger (like the bridge from Cyseal to Luculla or a cave entrance) that sends you to another zone or even a whole other level.
For texturing, take advantage of the clamp setting a lot. This basically limits the amount of texture that's applied. Also, don't underestimate the power of meshes (like grasses and whatnot) and objects like trees and rocks to make a scene look good.
An atmosphere can also help make a scene look a lot better. The default atmosphere has shadows off, and turning those on (and changing the pitch so there's longer shadows) can improve the look of a scene a lot. If you'd like to add a day night cycle, I'm working on that and will be fleshing out a good tutorial in my Day/Night cycle thread.
Ultimately, terraining is a lot of work. Expect to put in a lot of hours getting good at it, but when you finally do, it can be extremely rewarding. The D:OS engine is quite powerful and will only improve as Larian adds more assets and improves it, which is guaranteed since they're making two more games with it.