5 ways to make a GUID or UUID in ArcGIS

{60E263FD-DBDA-443D-83A5-8F79E96AFA76}
{032BB097-9EF9-45EB-8B54-3C191E4E9612}
{9C6B8BD1-C294-4C1A-8E76-96D9B2378E53}
{ED08D59B-0E47-4178-B469-72759B6CDF07}
{FF378680-36D6-4FAC-8419-0F12C77B7E28}

Globally Unique Identifiers or Universally Unique Identifiers: Humanity’s attempt to make enough unique values to assign one to everything on the Earth. These funny values are the subject of many debates on when to use them and when not to, should we use a GUID or an Integer? What is faster? Can a user remember a GUID (no, no they can not)? etc.

Side note: GUIDs and UUIDs are the same thing, a GUID is just Microsoft’s implementation of a UUID!  For a quick primer on GUIDs and UUIDs I suggest the Wikipedia pages.

We may not agree on when we should use them, or how to pronounce them (GUID seems most commonly pronounced GOO-id, but I don’t say it that way…) we can all agree that they are becoming more a part of our every day lives when dealing with data. As Data Geeks we often want to shield our users from these internal values, but we need them, and generating these values is the subject of much consternation but ultimately we need not be afraid, they are just 128-bit integers and there are simple ways to generate them in most programming languages!

Enough with the discussion, lets see how to generate these things! I picked 5 common languages that I have needed to generate GUIDs in for one reason or another, feel free to skip down to the one that matters to you: ArcMap/Pro (Python), Javascript, SQL, .Net, Swift.

ArcMap/Pro (Python):

Python has this handy library called uuid, we can easily convert this to a string using: "{" + str(uuid.uuid4()).upper() + "}"

Bonus: You can use the above method in a Calculate Field Code Block and simply call the function in the Value box.  For examples on using Esri’s Calculate Field tool, check out: http://resources.arcgis.com/en/help/main/10.2/index.html#//00170000004s000000

For a complete working example, check out this fiddle: http://pythonfiddle.com/makerandomguid

Javascript

Javascript can be a bit harder to generate a GUID in, however if we are using Esri’s Javascript library, then we have access to the Dojo library already, and we might as well use it! We can require['dojox/uuid/generateRandomUuid'] and from there convert to upper case and add brackets: "{"+uuid().toUpperCase()+"}" Putting it all together:

For a complete working example, check out this fiddle: https://jsfiddle.net/cmq3u7jn/

SQL

It seems that automating and building things in SQL is sometimes forgotten, but SQL provides a very powerful platform to store an analyze data, and there is always room to balance what you process in the database vs what you process on the client.  For example, when doing an insert via SQL you can easily generate the GUID that you need.  This example is for SQL Server, but Oracle is pretty close (try using sys_guid).

For a complete working example, check out this fiddle: http://sqlfiddle.com/#!6/9eecb7/1474/0

.Net

In a typically simple yet complicated fashion, .Net (c#) makes it pretty easy to generate a guid, it comes down to one line:  System.Guid.NewGuid().ToString("B").ToUpper();

For a complete working example, check out this fiddle: https://dotnetfiddle.net/TdPCoD

Swift

Last but not least, Swift.  Apple’s newest excursion into building new programming languages provides a pretty clean and neat syntax and is pretty fun to use!  Generating GUIDs in Swift is as easy as I would expect:  "{"+NSUUID().UUIDString+"}"

For a complete working example, check out this fiddle: http://swiftstub.com/15398662/?v=gm

What to do about “[Shape_STLength__]” Errors?

Oh boy….
Database error: The field is not nullable. [Shape_STLength__]. The geometry is not Z-aware.

TL;DR
That field shouldn’t be there, you can safely remove it. I wrote a quick script to do so here: https://gist.github.com/morehavoc/3e86f535bf204b040ee2

Now for the full story…

It took me a bit to figure this one out, I thought “Hmm, that looks like an Esri field, why would I be populating the Length from my javascript app, shouldn’t the server take care of that for me?” As it turns out I was wrong, that is not an Esri field… but in my case it was caused by Esri.

For me the field was in a feature class that I had created via a Python script using another feature class as a template; that was the issue. Esri, being so helpful, decided to convert the Shape.STLength() field into Shape_STLength__ and to top it off, make it required as well. The field Shape_STLength__ has no use, other than to cause trouble. The Shape.STLength() field is not actually a real field, it is a alias created by Esri to get the length of the geometry, handy right!

Once I figured out that was the issue, I was able to remove all of the Shape_STLength__ and Shape_STArea__ fields using a python script (making sure to shut down my services first) and all was well once again.

You can check out the python script here: https://gist.github.com/morehavoc/3e86f535bf204b040ee2

 

OAuth 2.0 Explained by Visiting Grandma

After my last post, I realized that it might be kind of hard to understand if you didn’t already have a basic understanding of authentication online.  To that end, I came up with a different way to try to explain how OAuth 2.0 works, by using my Grandmother as an example.

Lets say I am going on a trip to visit Grandma, when I get to her house I would like to have access to it so that I can see her and talk with her a while, but I don’t have a key!  In order for me to get into Grandma’s house I need to ask for permission, this is where OAuth2 comes in.

In this case I ring the doorbell and wait for Grandma to ask “Who is it?”  I respond, “Your Grandson Christopher” to which she replies “I’m coming!”  This is similar to the way that OAuth2 works, when I (The application) try to access data (Grandma’s house) I first have to ask permission from the user (Grandma).  Here Grandma acknowledges that she will let me in and lets me know that she is on the way just like the OAuth2 sever returning me a code.  The application can use this code to complete activation.

When Grandma gets to the door she can tell it is me by looking through the peep-hole in the door.  In OAuth2, this is where I would send my secret key and the code the server gave me to the server so that it can verify who I am.  In Grandma’s case all she has to do is look out the window or door!

Once Grandma has confirmed my identity she lets me into the house just like in OAuth2, once the server has confirmed who I am it passes me a bearer token which I (the application) can use to access secured items on behalf of the user.

If you want to learn more about OAuth, I listed several resources in my previous post which details more of the technical side of how OAuth2 works.

OAuth2 and ArcGIS Online

For a while now ArcGIS Online has had the ability to allow a user to login, and create a Map, publish data, etc.  And of course there were APIs that allowed us developers to get access to that data created by the users.  However, there was a big problem: The app couldn’t access the data without the user logging in, and to do that the app basically had to ask for the user’s username and password, which is a big no-no in the security world, and a level of responsibility that you probably don’t want if you were the developer.

Around March of this year Esri released an update to AGOL that for the first time provided a new way to authenticate users, OAuth 2.0.  And to make things even better, they didn’t just make up some new protocol  they used an industry standard and accepted process for authenticating users.  If you have already done OAuth 2.0 in the past, you can probably stop readying here and jump over to developers.arcgis.com and check out the walk-throughs and examples that Esri provides.  If not then here we go!

In a nut shell OAuth 2.0 is a standards based authentication system that provides reasonable protection, yet is simple and easy to implement correctly and allows developers to authenticate users without ever knowing that user’s password.

OAuth 2.0 is all about the workflow!  And it looks something like this:

  1. The Client or Application offers the user an option to login via AGOL.  This option is in the form of a link or button that directs the user to a web page hosted by AGOL.  Here the user has the option to login this providing the application access to his/her data. During the “redirect” to the AGOL login page the application identifies itself via a special ID, this allows AGOL to let the user know what application is requesting the login.
  2. Once the client logs in, AGOL sends the user back to the application, and provides that application with a special code.  This code is an agreement between the server and the application to allow the application to complete the login process, however the code itself does not allow access to any secured data.
  3. The application makes a POST request to AGOL that specifies the code, the application id and the application secret.  This secret is known only by the application and AGOL, not the user.  AGOL can then verify that these three pieces of information all match, if they do…
  4. AGOL returns a bearer token to the application.  This bearer token provides access to data via any AGOL API call that accepts a “token.”
  5. The application can now use the token from the server to request secured resources
  6. The server uses the token to validate the user and apply any permissions to requested resources, and returns either an error condition stating the user does not have access, or the data back to the application.

And authentication is complete!  The client can now use the bearer token to requested secured services, as seen in parts E and F above.  It is a little bit complicated at first, but there is plenty of code out there to help get you started.

There are variations on OAuth 2.0 that I did not cover here in this post, if you would like more information check out these links for more info:

One line if statements in Python

Every once in a while it is kind of nice to put a one-liner if statement in a Python program.  I don’t use these very often as they can make the code harder to read.  However I do think they have their place.  Mainly I use these when I need set the value of a variable based on a condition.  So lets say I start with a variable called dummy, and set it to None.

[python]

dummy = None

[/python]

And I want to set dummy to something, but only if dummy is still none, so:

[python]

dummy = (something if dummy is None else dummy)

[/python]

This is the same as writing:

[python]
if dummy is None:
dummy = something
else:
dummy = dummy
[/python]

The one line expression is a much simpler way to express this condition.

Installing MYSQL-python on Mac OS X

I like to do my web development on my MacBook; I find it much easier than working on my Windows machine.  Sometimes things don’t go so well when installing things on my Mac, at least not as easy as they are when installing on Linux.  One of these such things is installing MYSQL-python into a virtual-env for use in my Django apps.  When I run it usually get something like this:

[code]

(thegame)strix:online_app cmoravec$ pip install mysql-python

File "setup_posix.py", line 25, in mysql_config

raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

[/code]

The answer is actually pretty easy, before running pip (or permanently if you wanted) run:

[code]

export PATH=$PATH:/usr/local/mysql/bin

[/code]

This assumes that your mysql is installed in /usr/local/mysql/bin.  I installed MySQL here to get the binaries and headers so that I could install things that depend on them, I actually use MAMP to run my database.  I understand that if you use the bitnami MAMP then you don’t have to go through this trouble, oh well, Live and Learn!

The most usefully link for information about this is from stackoverflow.

Bitly Data API

Over my Christmas vacation I decided to take on a small programing challenge.  I had stumbled upon the Bitly api a few days earlier and noticed that it provides some information about location of links, etc. so I decided to give it a go!

After looking over the api and data returned, I decided that it would be cool to build a map that displays trending locations on Bitly.  My first task was to see if I could find a python api client library for bitly, since python was going to be my server language of choice.  After looking over the apis listed here, I found that all of the apis were centered mainly around shortening urls and I needed more than that! So I picked up my favorite python rest library (restlib) and started working. I only needed a few api calls, but I figured it would be nice to build a simple library out of it so that I could reuse it.

I started out by creating a Connection class that would contain all of the methods to connect to the api.
[python]
class Connection():
“””Class that represents a connection to the bitly api.”””
access_token=””
_baseUrl=””
def __init__(self,access_token):
“””Creates a connection object with the specified access_token. This token should come from an oauth2 process.”””
self.access_token = access_token
self._baseUrl = “https://api-ssl.bitly.com/v3/”
[/python]

This gives me a simple framework that I can use to create a “connection” to the api, and ask for specific information, without trying to track an access_token though different requests. My first method was to find “hot phrases” as defined by the bitly api /v3/realtime/hot_phrases. This returns a list of trending phrases right now, and the links that are connected to them.

[python]
def realtime_hot_phrases(self):
“””Lists the current realtime hot phrases.”””
url = self._fullUrl(“realtime/hot_phrases”)
params={“access_token”:self.access_token}
result = restclient.GET(url,params)
result = json.JSONDecoder().decode(result)
self._checkStatus(result)
return result[“data”][“phrases”]
[/python]

In order to make my life easier I created a couple of private helper methods to make my life easier. The first is, “_fullUrl”, it takes in the url component that this particular method uses, and returns the full url. This just makes it easier to manage the urls, and I didn’t have to duplicate string concatenation all over the code.

The second is, “_checkStatus”, this method automatically checks the result returned by the http call for any errors that may have been returned by the api.

[python]
def _checkStatus(self,results):
“””Checks the status of a return, and raises Exceptions as appropriate.”””
try:
if results.has_key(“status_code”):
if results[“status_code”] == 200:
return
else:
raise BitlyError(“Status Code is not 200 (%s) (%s)”%(str(results[“status_code”]),str(results[“status_txt”])))
else:
raise BitlyError(“No status code found.”)
except Exception as e:
raise e
[/python]

This function automatically looks for the status code attribute of the result object, if it exists, and it is 200 the function will just return. If the status code is not 200, the a BitlyError is raised with details about the status that was returned. One thing that would make the library a bit more usable would be to add more status code errors, and raise errors with more detailed information about what happened.

With that done, you can easily get the hot links from bitly like this:

[python]
bit = Connection(“”)
print bit.realtime_hot_phrases()
[/python]

I added a few more methods for things like:

  • link_countries – To get the countries that a link is related to
  • link_locations – To get locations (a bit more specific than country) that a link is being used from
  • link_history – Gets the current logged in user’s link history
  • realtime_bursting_phrases – Gets  a list of phrases that are bursting in popularity among links right now

You can download the library here if you want to try it out.

I will add another post later about the final product LinkMap!

Migrating to DreamHost

Just finished moving my site from godaddy.com to dreamhost.com.  So far the transfer has been pretty painless, but all I did was migrate the hosting!

After getting setup, I started working on a project to put some Bitly API data on a map using Django/Python.  I quickly discovered that the version of python on DreamHost was too old, so I set out to install my own.  I found these instructions to be the best: http://www.ricksresources.com/2012/07/install-python-27-dreamhost/

Once you can SSH into your DH server, do the following:

[bash]
[paramaribo]$ mkdir python
[paramaribo]$ cd python
[paramaribo]$ wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
[paramaribo]$ tar xjf Python-2.7.3.tar.bz2
[/bash]

That will download python for you, and extract it.  Now you will need to build Python:

[bash]
[paramaribo]$ cd Python-2.7.3
[paramaribo]$ ./configure –prefix=$HOME/python
[paramaribo]$ make
[/bash]

After running this, you may see some text like this:

[bash]
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _tkinter bsddb185
bz2 dl imageop
sunaudiodev
[/bash]

These Python libraries probably won’t be needed, so don’t worry that they didn’t build. Now run the following to install Python:

[bash]
[paramaribo]$ make install
[/bash]

Once that is done, python should be operational! Run the following to add your new python to your PATH so that this python will always run:

[bash]
[paramaribo]$ echo ‘export PATH=$HOME/python/bin:$PATH’ >> ~/.bash_profile
[/bash]

Restart your SSH session, and try running “python” at the prompt, and you should have python 2.7.3 all up and running! The next step is to set up a virtual environment and get some stuff running.

A python virtual environment lets you have isolated versions of python running with different installed libraries. This makes it really easy to setup different applications on the same server even when they need different versions of python, or different libraries installed.

[bash]
[paramaribo]$ wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.8.4.tar.gz
[paramaribo]$ tar xzf virtualenv-1.8.4.tar.gz
[paramaribo]$ cd virtualenv-1.8.4
[paramaribo]$ python setup.py install
[/bash]

I keep all of my environments in an env folder in my user’s root directory:

[bash]
[paramaribo]$ mkdir env
[paramaribo]$ virtualenv env/linkmap.moravec.net
[bash]

Now, switch to your new environment, install Django, and mysql drivers:

[bash]
[paramaribo]$ source ~/env/linkmap.moravec.net/bin/activate
(linkmap.moravec.net)[paramaribo]$ pip install Django
Downloading/unpacking Django
Downloading Django-1.4.3.tar.gz (7.7MB): 7.7MB downloaded
Running setup.py egg_info for package Django

Installing collected packages: Django
Running setup.py install for Django
changing mode of build/scripts-2.7/django-admin.py from 664 to 775

changing mode of /home/cmoravec/env/linkmap.moravec.net/bin/django-admin.py to 775
Successfully installed Django
Cleaning up…
(linkmap.moravec.net)[paramaribo]$ pip install MySQL-python
Downloading/unpacking MySQL-python
Downloading MySQL-python-1.2.4c1.zip (113kB): 113kB downloaded
Running setup.py egg_info for package MySQL-python
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
Extracting in /tmp/tmpZZ7doS
Now working in /tmp/tmpZZ7doS/distribute-0.6.28
Building a Distribute egg in /home/cmoravec/env/linkmap.moravec.net/build/MySQL-python
/home/cmoravec/env/linkmap.moravec.net/build/MySQL-python/distribute-0.6.28-py2.7.egg

Installing collected packages: MySQL-python
Running setup.py install for MySQL-python
building ‘_mysql’ extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Dversion_info=(1,2,4,’rc’,5) -D__version__=1.2.4c1 -I/usr/include/mysql -I/home/cmoravec/python/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o -DBIG_JOINS=1 -fPIC
gcc -pthread -shared build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib/mysql -lmysqlclient_r -o build/lib.linux-x86_64-2.7/_mysql.so

Successfully installed MySQL-python
[/bash]

Once that is done, you should be good to go!

For details, try: http://www.ricksresources.com/2012/07/writing-wsgi-apps-on-dreamhost-shared-hosting/