6.1 Objectives

This Lab contains 2 parts. In Part A you will:

1. Create a high resolution Digital Elevation Model and hillshade from airborne LiDAR LAS files
2. Create new feature classes in your WMA geodatabase to store field observations;
3. Export your ArcMap project to ArcGIS Online for upload to the Collector App on a field tablet computer;

In Part B, to be done during the following lecture period, you will:

1. Learn procedures for capturing GPS points, lines and polygons with the Collector App;
2. In pairs, practice with a iPad Mini 4 tablet capturing the locations of polygons and lines on the Campus Main Building Mall.

6.2 The Problem and the Data

The goals of our weekend field trip are to three fold:

1. Map the distribution of Precambrian rock outcrops.  These cannot be entirely mapped from areal photographs because of masking by vegetation, but are more easily recognized from a "bare earth" Lidar terrain model, which you will create below.  The digital outlines of outcrop polygons allows for, among other things, an accurate estimate of the area underlain by vegetation vs. rocks within some of the WMA pastures.  This is an important statistic for game management studies.  Subtracting the sum of the rock polygon areas from the total pasture areas will give this statistic.

2. Map and measure the orientation of fracture/joint sets and planar dikes in granite.  Note relative ages where possible.  These data are needed to better address questions about the age and orientation of stresses responsible for extensive fracturing near the southern margin of the Katemcy Pluton.  Our high resolution airborne Lidar terrain models will reveal a pattern, but field data are required to interpret it.

3. Collect observations (principally geo-located photos) that bear on the emplacement and cooling history of the granite.  Are there patterns to subtle variations in texture, grain size, dike abundance and orientation, etc.?  With the ability to collect observations at a high spatial resolution it is possible to answer these questions.

   Data 

The GIS lab portion (Part A) uses the following data:

• UTM zone 14 NAD83, LAS-format, airborne LiDAR files acquired during flights in 2007, provided by the Texas Natural Resources Information Service (TNRIS).  These were ordered and collected from their Austin office for the cost of a 50 Gb flash drive;
• a UTM zone 14 NAD83, 0.5 meter-resolution, 2015 digital orthophoto (NW quarter of the Purdy Hill Quad.) from TNRIS in jp2 format;
• a UTM zone 14 NAD83, ~10 meter-resolution hill-shaded elevation model, stored in ESRI grid format, derived from National Elevation Dataset data.

**Download the Lab_6_data folder to your flash drive, NOT YOUR NETWORK STORAGE.**  This file contains over 4 Gb of data and may take up to 5 minutes to download.

6.3 Working with LiDAR Data in ArcGIS 

LiDAR data are fundamentally clouds of points ("point clouds") with XYZ coordinates and attributes. They are commonly stored and distributed in LAS (LASer) format files, a non-propriety format that has gained wide acceptance in recent years.  ArcGIS has extensive tools for importing and working with LAS files.  Our goals for this part of the lab are to:

• Import LAS files into ArcGIS and examine their properties
• Create a raster digital terrain model ("DTM") from these vector files

1. After checking to be sure that the Spatial Analyst and 3D Analyst Extensions are checked on in a blank ArcMap document, open the ArcCatalog window inside of ArcMap, right-click on your Lab_6_data "LAS_WMA_2007" folder and select "New", create a "LAS Dataset" and rename it MMWMA.lasd.  This creates a container into which we can import LAS files.  A "LAS Dataset" (.lasd) has special properties that allow us to examine some of the unique features of LiDAR data and is required for some LAS processing tools.
   
   
2. Add the six LAS files (each is a tile of a single large area and has a file name that ends with .las) to this new LAS Dataset by right-clicking on the LAS Dataset icon, selecting "Properties...", the "LAS Files" tab and the "Add Files..." button (Fig. 3 below).

Figure 3. Adding files to a LAS Dataset from the LAS Dataset Properties window

3. Examine the Properties of the imported files. Note that each file contains 26 million points - we will be working with over 150 million points in this exercise (!) - that are spaced at about 0.63 meters (Fig. 4).  The maximum (Z Max) and minimum (Z min) elevation (in meters) of the points in each file are also listed.

Figure 4. LAS Dataset Properties window, showing point counts, spacings and minimum and maximum elevations for 6 airborne LAS files.

4. Open the "Statistics" tab and click the "Calculate" button - this may take some time... there are over 150 million points(!)
   
   
5. As shown in Figure 5, the result provides Statistics on two important parameters.  This will be important later - pay attention as you read on...

Figure 5. LAS Dataset statistics, summarizing statistics for all six of the LAS files in Figure 4.

LiDAR data consist of multiple "Returns" (upper left table in Figure 5): light pulses that bounce back to the instrument. A "return" is a specific arrival at the instrument from a single laser pulse.  For each pulse, these are classified by whether they return quickest ("First returns") or at successively later times (Second, Third, Forth, etc.).  As shown in the statistic table, of the 158,393,951 returns, about 50% are first returns (also classed as "First of Many") and 50% are later ("2nd", "Last", "Last of Many").  More recently acquired airborne data will often contain a more detailed classification of returns - these 2007 data, in LAS 1.1 format (see "Classification Codes" table of Figure 5), contain only two classifications, making them less useful for some applications.

During post-processing of raw data and during conversion to LAS format, most returns are assigned a standard Classification Code (middle table of Figure 5) based on the origin of the return.  The distinction between return number (i.e first, second, etc.) and Classification code is important - it is easy to understand that for partially vegetated areas some first returns will come from the ground whereas others will from the tops of trees.  We are interested in returns that are classified as ground returns - coded 2.  For this dataset, points with ground returns are least abundant (~21% of all data); note that all other returns (~79%) are "Unclassified" (code 1; again see middle table of Figure 5).  Again, more recent datasets commonly contain classification for 4 or more additional categories: i.e. water, vegetation, buildings, etc.

The strength or Intensity of returns (middle table) is also measured (on a scale of 0 to 65,535) - some pulses come back strong, some weak.  Strong returns come from highly reflective surfaces, weak from surfaces that absorb some of the laser pulse.  Minimum and Maximum Intensity measurements are listed for each classification.

6. Click "OK" to store the statistics. 

7. Drag the MMWMA LAS Dataset from ArcCatalog into a blank Arc Map window.  You have just added the entire point cloud, all ~158 million points!

8. ArcMap shows only the red outlines of each of the six data tiles (this keeps redrawing times reasonable) until you zoom to a specific area.

9. Zoom into the lower right corner (to a scale of 1:10,000 or greater) of the upper middle tile.  Your screen should look similar to image below.  You are looking at a two dimensional view of a point cloud, color coded by elevation (Figure 6) - the ArcMap Table of Contents shows the range of elevations (in meters) represented by each color.  White represents areas of no data.

Figure 6. Point cloud data, color coded for elevation.

10. Open the LAS Dataset toolbar (Customize>Toolbars>LAS Dataset), which contains a variety of options for viewing and manipulating point clouds.



If the toolbar is grayed-out, you will have to turn on the Spatial and 3D analysts extensions, as indicated in the Step 2 above.

11. Want to see something astounding? Use the Profile Tool (second from right on the LAS Dataset toolbar) to construct a short vertical profile (a cross section) through the point cloud (Figure 7).  A tool tip, visible when hovering the mouse over the tool, explains how.  These are unfiltered points, so we are viewing returns irrespective of Code Classification (tree tops clearly visible!).  We won't spend much time with this tool but it's just too cool to ignore!  Explore the other tools on this toolbar - they provide very powerful ways to view and interactively filter LAS point clouds.  If they don't seem to work, try sampling a smaller area.

Figure 7. Vertical profile (a.k.a. cross section)created with the Profile Tool of the LAS Dataset toolbar through a small part of the unfiltered point cloud .  Elevations are color coded and ground versus vegetation is clearly distinguishable.

12. Can't resist one more trick with this toolbar... create a surface elevation model (a TIN; Figure 8) with the Elevation tool on the toolbar by zooming in or out of your Map.

Figure 8. TIN of point cloud data.  The surface roughness is a consequence of vegetation, mostly oak and juniper trees.

The maps/profiles created with the LAS Dataset toolbar are visualizations created on-the-fly, not permanent products.  As such, they are of limited use for analysis.  We would like to create a permanent raster dataset from this LiDAR (vector) point cloud - specifically a high resolution, "bare earth" Digital Elevation Model (DEM; nomenclature varies - LiDAR-derived bare earth rasters are commonly referred to instead as Digital Terrain Models (DTMs)).  This will require a tool from ArcToolbox.

13. Using the Search window in ArcMap, search "LAS to Raster" for the tool needed for this conversion.
    


14. IMPORTANT - Before running this tool make sure a) that you are zoomed completely out and can see the red outlines of all of the data tiles in ArcMap; b) that the LAS Dataset Toolbar has the "Filters" tool drop-down set to "Ground" and the Point tool drop-down set to "Elevation", as shown below.  These choices will be recognized by the "LAS Dataset to Raster" tool in ArcToolbox if we take care to choose the ArcMap LAS Dataset layer as the Input Dataset and not browse to and choose the original LAS Dataset.

Figure 8a.  The LAS toolbar, showing the Point tool dropdown open and "Elevation" highlighted.

15. Open the "LAS Dataset to Raster" tool from ArcToolbox or the Search window.  If not already shown, show the Help for this tool. You may also need to enlarge the window to see all options, as seen below. USING THE DROP-DROWN ARROW AND NOT THE FOLDER ICON (see Figure 9 below), select MMWMA.lasd as the Input the LAS Dataset.

Figure 9.  The LAS Dataset to Raster tool.

• name your output raster MMWMA_DTM, to be stored in your LAS_WMA_2007 folder of Lab_6_data folder
• Value Field is ELEVATION,
• Interpolation Type is Binning with"MINIMUM" (this is a bare earth model...) as the Cell Assignment Type and NATURAL_NEIGHBOR as the Void Fill Method.
• DO NOT CLICK OK YET -we need to specify the raster resolution (cell size), which requires some thought.  The raster cell size should never be smaller than the point spacing of returns.  As seen in the statistics, the average return spacing is about 0.6 meters. This is for all returns, irrespective of classification.  We are using Class 2 (ground returns) only, not all points, so the spacing for these returns may be significantly greater (3x to 4x greater- not all points will have ground returns).  A conservative approach for a bare earth DTM is to set the raster cell size to ~ 3 times the average return spacing.  For our data this means about a 2 meter cell size, SO SET THE "Sampling Value" TO 2.
• Z Factor is 1. Click OK. This will take a few minutes...

If all went well your DTM should look like the Figure 10 below.

Figure 10. Digital Terrain Model (DTM) created from six LAS files using the LAS Dataset to Raster tool.

This new DTM will be easier to visualize in shaded relief.

16. Using the Search window, find the Hillshade tool in the Spatial Analyst ArcToolbox and create a Hillshade.  Your Hillshade should resemble Figure 11 below.  Take some time to explore this at higher magnification - it's an amazing product!
    Note:  If the tool gives an error, do use the default file storage locations but instead save it to a flash drive!  Likewise, if the source DTM file is located in network storage, try copying (in ArcCatalog) to a flash drive and use that source DTM instead.
    
    
17. Load the hillshade_sm file into your ArcMap project.  hillshade_sm is a ~10m resolution hillshade created from a National Elevation Dataset (NED; recall the lecture on Elevation Models) 10m DEM.  NED data are not derived from Lidar measurements.  As illustrated in the comparison below (Fig. 12) and from your own observations, the difference is dramatic.  Fracture patterns that are, at best, obscure in the ~10m data are clearly visible is the 2m DTM.

Figure 11.  Hillshade of the Lidar DTM.

Figure 12.  Comparison of LIDAR-derived, 2m DTM and the conventional 10m, NED-derived hillshade.  Note the dramatic difference in detail.  A lake and road are clearly recognizable near the bottom of the 2m image on the left but unrecognizable on the right.

The 2m DTM hillshade exceeds the boundaries of the WMA and is a ~56 Mb file stored in uncompressed ESRI grid format.  We would like a smaller compressed file for our handheld field units.  We can make one by clipping the file to the WMA boundary and converting the result to a JPEG image.  To do so:

• Load the "Property_Footprint" file into ArcMap;
• Use the "Extract by Mask" tool in ArcToolbox to clip the DTM hillshade to the WMA boundary polygon.  In so doing, you will have to choose a file name less than 13 character; I used "clp_hshade" for my clipped version.
• This new file is also in ESRI grid format but is now ~32 Mb (verify this examining the file Properties>Source tab).  This file is still too large.  To compress this new raster:

1. Right click on the file name in the ArcMap table of contents, select "Data">"Export data..." to show the following Export Raster window:



Fill in the fields with the values shown in the red box above, making sure you save this to a place where it can be easily retrieved, and click "Save".  This will create a 2m resolution, JPEG compressed raster, saved in Geotiff format.  Through these processes, we have reduced the file size from ~56 Mb to about 8 Mb without appreciably affecting the utility of the hillshade!  This new Tiff file is now small enough to store and quickly render on our handheld units.

6.4 Constructing a Map for Field Work

The field data we will collect includes the outline of outcrops and dikes (lines or polygon features), joint measurements (point features), dike orientations (point features) and photos (tied to points). You made a geologic map with base map layers in Lab 4 and 5 that will provide context for your field work.  The LiDAR hillshade created above will do the same. What remains is to create new feature classes within the feature dataset (Geology) of the geodatabase (WMA_Map_XXX)you created in Lab 4 that will store your field data. You have already done many of the steps below in Lab 4. Refer to it if you've forgotten how to create feature classes and domains.

1. Open ArcCatalog and browse to the Geology feature dataset of your WMA_Map_XXX geodatabase in your Lab_4_and_5_data>My_data folder.
    
2. Time to create the empty Feature Classes that will contain the GPS-derived points, lines and areas...

1. Right-click on the Geology Feature Dataset icon, select "New...", then create new Polygon, Line and Point Feature Classes (named Polygon_XXX, Line_XXX and Point_XXX, where XXX is yourinitials). Click "Next" and move on to step b below;

Figure 13. The window for creating a new polygon feature class.
 

2. The polygon feature class will store GPS-derived outlines of granite outcrops and any features that are polygons.  We need an attribute field that records the feature being mapped (e.g. "granite", "pegmatite" or “other”) that can be entered as we collect the data.  So... add two Text fields to the polygon feature class, one called "FEATURE" and another called "COMMENT". See section 4.34 in Lab 4 if you've forgotten how. The length of the FEATURE field should be 9 and the COMMENT field 30.  Leave all other Field Properties blank for now.  Please use these precise field names, including capitalization, for this and all other feature classes. Merging and appending files from different field units is much easier if everyone uses exactly the same field names and properties.  Click "Finish".
   
   
3. Add a Domain to your geodatabase (by right-clicking on your WMA_map_XXX geodatabase icon, then Properties...) called PLY_TYPE, (Field Type is Text) that is a coded-value domain containing the coded values of "granite", "pegmatite", and "other" (see Lab 4 if you've forgotten how).  Recall that domain names and coded values are case sensitive - use upper and lower case letter that exactly match those given here!
   
   
4. Attach this domain to the polygon attribute field FEATURE (again see Lab 4.)
   


5. Repeat Step 2a above again to create a line feature class called "Lines_XXX", being sure to change the "Type" to line.
   


6. The line feature class will be used for outcrop outlines that can't immediately be seen to close on themselves (i.e. can’t be mapped as polygons) and dikes that are too small to map as polygons. The attributes (within new Domains) and the new fields to create are:
   
   
   1. 9-character text field, called  "FEATURE", that will contain coded values from a text Domain called "LN_TYPE" of "Dike", "Outcrop", "Contact", "Qtz_vein" (note this change on 10/14/2019), "Fault", and "Other".
   
   
   
   2. 8-character text field, called "SYMBOL",  that will contain coded values from the "Exposure" Domain (exposed, inferred, covered) created in Lab 4.
      
   
   
   3. 30 character text field, called "COMMENT", without an attached domain.
      
   
   
   4. 3 character text field called "AUTHOR", with no domain but a default set to your initials.
      
   
   
   5. Create these new Fields and their Domains with the above coded values AND attach the Domains to the Fields, as in steps b, c and d above.
      
      
   6. Repeat Step 2a again to create a point feature class, being sure to change the "Type" to point.
      
   
   
   7. The point feature class will be used to record features too small to be polygons, and for strike and dip measurements and photographs. We will need fields for:
      
      
      1. 10-character text field, called "PT_TYPE", that will contain coded values from a text Domain called "PT_TYPE" of "Joint", "Fault", "Foliation", "Qtz_vein"(note this change on 10/14/2019), "Dike", and "Other".
         
      
      
      2. 3-character long integer field, called "STRIKE", that will contain coded values from a long integer Domain called "strike" of every third integers between 0 and 357 (i.e. Codes of 0, 3, 6, 9, 12 etc. with Descriptions of 000, 003, 006, 009, 012, etc. to 357; yes, all 120 values).  This can be create by importing a table created in Excel and saved as .CSV or .TXT format using the "Table to Domain" tool in ArcToolbox.  Domains created from tables imported this way will be long integer, coded value domains, so the domain "field type" must be "long integer' for this import technique to work.
         
      
      
      3. 2-character long integer field, called "DIP", that will contain coded values from a long integer Domain called “dip” of every second integer between 2 and 90 (i.e. Codes of 02, 04, 06, etc., with Descriptions of 02, 04, 06 etc.; 44 values in all). This can be create by importing a table created in Excel and saved as .CSV or .TXT format using the "Table to Domain" tool in ArcToolbox.  Domains created from tables imported this way will be long integer, coded value domains, so the domain "field type" must be "long integer" for this import technique to work.
         
      
      
      4. 5 character text field called "PHOTO", with an attached coded value domain call TrueFalse with coded values and Descriptions of "True" and "False"
         
      
      
      5. 30-character text field, called "COMMENT", without an attached domain.
         
      
      
      6. 3 character text field called "AUTHOR", with no domain but a default set to your initials.
         
      
      
   8. Create these new Fields and their Domains with the above coded values and attach the Domains to the Fields, as in steps c and d.
      
      
9. Symbolize your three feature classes to match the figure below and save each as a layer file that can be used in subsequent uses of these feature classes.
   
   
   
   
10. Test your feature classes by successively editing each - creating a new feature for each and assign it attributes.  If you do not see a drop-down menu when assigning an attribute then your have not attached your domains to the feature class(es).  Correct this mistake and try again.   Do not save your edits.

What about the raster files?  The Lab_6_data folder contains a very high resolution DOQ and our newly created Lidar 2m hillshade raster; should we import these into the geodatabase? In this case the disadvantages of doing so outweigh any advantage. In particular, the color DOQ is a large jp2 format file that would get much larger when uncompressed and stored in IMG format, which is the format required by the geodatabase. There is no real advantage to doing this, other than having everything in a single container, and we are left with a file that is >150 Mb. We could instead create a geodatabase raster index (see Help files on this topic), but for the few rasters we will work with this also provides no real advantage. We will keep the rasters separate from the geodatabase for these reasons.

Congratulations, you have now completed the database you will need for this project.

OPTIONAL 6.5 Making a Layout for the Avenza Map App - Creating a GeoPDF OPTIONAL

1. Open your completed ArcMap from Lab 4&5.  Add the Lidar hillshade created above.
   
   
2. Load your empty polygon_XXX, line_XXX and point_XXX feature classes you just created and move them to the top of the Table of Contents if not already there.
   
   
3. Order the remaining layers so that the Hillshade is at the bottom, the DOQ is second from the bottom, and all remaining layers above these.
   
   
4. Set the Display Properties of the rock units and the outcrop polygons to 50% transparent.
   
   
5. Zoom to the WMA boundary layer and SAVE THE MAP document to your Lab_6_Data folder.
   
   
6. Switch to Layout mode and make a map with a 50 meter UTM grid, scale bar, north arrow, name, etc.  Go to "Page and Print Setup" and uncheck "use printer paper settings" and Set Page to Width: 36" and Height to 40".  Got to layout mode, adjust the edge of the WMA to the edge of the data frame to fill as much as possible to the layout boundaries.  We will make one map for display in the  Avenza Map App on the iPads.  This will show the LiDAR hillshade layer turned on with other layers partially transparent above it. Once the layout is satisfactory, go to File>Export Map>, Save as Type PDF, resolution 200 dpi, Image quality Best, Advanced tab> check on "Export Map Georeferencing Information".  This will create a GeoPDF that can be used in the field with your GPS enabled IPad.

To get this PDF onto your iPad or other mobile device, it needs to be imported through the PDFMaps App.  To do so requires that the GeoPDF is in cloud account (an iCloud, DropBox, Google Drive or Box account will work).  If you don't already have one, create a (free) cloud account, being sure to keep track of your user name and password!  Place your GeoPDF in a cloud folder so it can be retrieved by your iOS or Android mobile device, and by one of our iPads.

Finally, for lab next week you will need an ArcGIS Online account.  Create one now following instructions  at UT GIS & Geospatial Data Services.  This will create an account (even if you already have one) that is linked to the UT Organizational Account, a necessary step to seamlessly share data and have unlimited storage.

You're done.

Mark Helper, Spring, Fall 2016, 2020