AlexDGlover Blog - Articles

What's pass-reference-by-value?

2020-08-14T00:00:00+00:00

I’m not an authority on this - take my explanation with a grain of skepticism and do your own research.

I’ve been working on going over some of the fundamentals of computer science recently since I don’t have a traditional computer science education. Pass-by-value recently came up, and I thought it would be a quick foray to properly learn the difference between pass-by-value and pass-by-reference. Surely something so fundamental must have clear answers, right?

And then it turned into a huge fucking rabbit hole, with lots of dissenting views, and very confusing discussions. Just Google it, check out some of the StackOverflow articles. There’s a lot of content, but it’s not very clear. In the abstract, the concepts are simple. In the context of Python and Ruby, it’s a confusing mess.

So that’s what this post is about - trying to clarify the differences between the two and define what “pass-reference-by-value” actually means.

Quick, naive explanation

The TL;DR version of pass-by-value vs pass-by-reference is simple - pass by value means the value of a function parameter gets copied into the body of a function, but it’s not the same object in memory as the original parameter. You can’t mutate or change whatever you passed into the function, because your function doesn’t actually know about the original “thing” - it only has a copy of the value of the “thing.”

Pass-by-reference is the opposite. When you invoke a function with a parameter, the function body is referring to the same object in memory as the original. Any mutations of the object inside the function body will be persisted outside of the function body.

We can see an example of pass-by-value with a little Ruby:

my_string = 'foo'
puts "my_string is '#{my_string}'" # my_string is 'foo'

def mutate(some_string)
    some_string = 'not foo'
    puts "string inside function is '#{some_string}'"
end

mutate(my_string) # string inside function is 'not foo'

puts "my_string is still '#{my_string}'" # my_string is still 'foo'

You can do the same thing in Python, Java, or any other pass-by-value language and get the same result. And it makes perfect sense - inside the function, some_string simply has the value of foo, so if we assign it to something else it doesn’t affect my_string elsewhere in the function.

A contradicting example

Let’s do one more simple example in Ruby, to build confidence in our newfound understanding.

my_string = 'foo'
puts "my_string is '#{my_string}'" # my_string is 'foo'

def mutate(some_string)
    some_string.upcase!
    puts "string inside function is '#{some_string}'"
end

mutate(my_string) # string inside function is 'FOO'

puts "my_string is now '#{my_string}'" # my_string is now 'FOO'

Confused? Yeah, me too. This confusion is what spurred my research. We know Ruby uses pass-by-value, we just empirically demonstrated that as fact - so how is it possible that we could mutate an object inside of a function and have that change persist outside of the function?

For anyone who uses Ruby or Python, they might argue that it behaves like pass-by-reference. Other people would explain the behavior by saying Ruby and Python are “pass-reference-by-value” or “pass-object-by-value” but what exactly does that mean?

More layers to the cake

It turns out that there are really two layers to this behavior, and everything makes a lot more sense once you are familiar with them.

A language can support pass-by-reference or pass-by-value semantics, or both. For example, C# supports both On top of that, the type of parameters you pass into a function can be a value type or a reference type, regardless of whether pass-by-value or pass-by-reference semantics are used. This is what makes classifying languages so confusing - a language that uses pass-by-value semantics and reference types appears to behave like pass-by-reference.

Let’s go back to our Ruby examples. Ruby is always pass-by-value. However we saw earlier that it behaved like a pass-by-reference language. That’s because the value of the my_string parameter is a reference to an object in memory. The value is still being copied when the function is invoked (hence pass-by-value), but the value is a reference to the same object in memory.

Let’s do another quick example in Ruby to demonstrate this:

my_string = 'foo'
puts "my_string is '#{my_string}'" # my_string is 'foo'
puts my_string.object_id # 60

def mutate(some_string)
    puts some_string.object_id # 60
    some_string.upcase!
    # object_id will still be 60 - still referring to the same object in memory
    puts some_string.object_id # 60
    puts "string inside function is '#{some_string}'" # string inside function is 'FOO'

    # assigning a value to `some_string` creates a new object in memory, and the value
    # is updated to reference that new object in memory. it did not change the original
    # object in memory. because of this, the ID will be different
    some_string = 'not foo'
    puts some_string.object_id # 80
end

mutate(my_string)

puts "my_string is now '#{my_string}'" # my_string is now 'FOO'

For those who need visual help like me, here’s a series of diagrams explaining the same concept. In each digram, the code will be on the left and a model of the objects in memory will be on the right.

When we first initialize my_string with the value 'foo', a new object is created in memory that holds the string foo.

When we invoke the mutate method and pass in my_string, a copy of my_string is made. That copy has the variable name some_string, and the value of the variable is the reference to the object in memory (not the string foo). It’s a new variable inside of the function scope, but it refers to the same object in memory.

If we mutate the object inside the function body, it affects the object both inside and outside of the function scope.

Now here’s where the difference between pass-reference-by-value and true pass-by-reference is made clear. If Ruby were pass-by-reference, we could update my_string to refer to a new object in memory. But we can’t - we never actually had my_string within the scope of our function, just a copy of that variable that refers to the same object in memory. If we try to assign some_string to a new object, it has no effect on my_string outside of the function body. It just creates a new object in memory.

Compared to languages with pointers

Rust, Golang, C, and C++ all support pointers. Some people will say that this means you can write functions with pass-by-value or pass-by-reference semantics. Don’t believe their lies - strictly speaking, these languages are also pass-by-value.

In Ruby and Python, every variable is a reference to an object in memory. In Rust/Golang/C++, both references and pointers are supported. However the differences between pointers and references is subtle, and don’t affect the examples we used in this post. The ELI5 version is that references are like “implicit pointers” and don’t require extra syntax to de-reference them. We could implement the same exercises above in any of these languages, using references or pointers, and the behavior would be the same.

So the only real difference is that these languages require us to be explicit about pointers/references, whereas Ruby and Python use references implicitly.

Conclusion

In short, pass-reference-by-value is just pass-by-value with implicit references. Unless you’re using C#, Perl, or some exotic language, you’re more than likely using pass-by-value, either with explicit pointers/references (Golang, Rust, C++) or implicit references (Python, Ruby).

I hope this helps demystify the nature of pass-reference-by-value for you and spares you the StackOverflow rabbit hole. Here are some links for content I reviewed to learn about this, in case you want to do your own research:

Is Ruby pass by reference or pass by value?
my post is like the ELI5 version of this post. What I didn’t understand was that assignment was the key difference in behavior
StackOverflow - Is Ruby pass by reference or by value?
- This answer in particular made it more clear to me
C# vs Python argument passing
Value type and reference type
Variable References and Mutability in Ruby

Linkets - A Zendesk App

2017-10-29T00:00:00+00:00

It’s official! Linkets has been formally published on the Zendesk App marketplace - go check it out!

One of the coolest things about Zendesk (both the company and the software itself) is how developer-friendly it is. The API is well documented and powerful, and they even provide an App Framework that allows you to develop “widgets” that are displayed in the standard Zendesk UIs. Full disclosure, I currently work at Zendesk.

Zendesk Support has a ton of features, but doesn’t have the ability to link related tickets. There is a Linked Ticket app you can install for free, but this only allows you to create new tickets, and only as a child ticket. Want to link to an existing ticket? Too bad. Need to support relationships other than parent/child? Too bad. I want people to use and love Zendesk, so I decided to take a crack at my own version of the linked ticket app. I titled my app “Linkets.”

The concept for the app is simple. Linked tickets are simply tickets with extra metadata about one another. The Zendesk API supports reading and writing tags on tickets, and tags are an excellent way to store metadata. When you open a ticket, the widget reads the tags, parses any tags that are specific to links, looks up those related tickets and displays them in the widget.

Users obviously need to be able to create new links. To do so, users can simply search for tickets (using Zendesk’s search API, which is a speedy, badass, full-text search platform), click on the button to create the link, and then choose how the two tickets are related. Behind the scenes, this is another API call that updates the current ticket and the linked ticket.

That’s it. Linkets is very lightweight, won’t eat through your API limits, and is easily modifiable/extensible. Don’t like the built-in relationship types? Just modify the JavaScript for handling ‘relationship codes’ and the relationship codes set in the UI.

The app isn’t published on the Zendesk App Marketplace yet, but in the meantime you can download the ZIP file from the GitHub repo and install it as a “Private App”.

Linkets on GitHub

The Zendesk developer docs are so good, I won’t bother walking through how to write your own app. If you’re interested in writing your own app, check out the Build Your First Support App Walkthrough.

Let me know if you end up using Linkets, or if you’ve written your own custom app!

As always, I hope you found this useful.

DNSReRoute - A Full SaaS Platform

2017-05-21T00:00:00+00:00

Dec 28 2017 Update OpenShift deprecated their original offering, so the API backend is completely dead. All functionality in the single page app (besides the Auth0 integration) is now broken.

About a year ago, I had a use case for a simple redirect. I just needed to redirect developers from an old Jenkins DNS record (that pointed to an instance that was destroyed) to a new DNS record (pointing to the new, running Jenkins instance). Now this is a trivial task if you want to run NGINX or Apache, but becomes more challenging if you want to avoid that overhead. I tried hosting a simple JavaScript redirect using S3, but the bucket obviously didn’t match my DNS name, so that didn’t work. Route53 doesn’t support HTTP redirects. I was at a loss - how could something so trivial require a whole web server to accomplish?

I started looking into 3rd party services that could do this, mostly out of curiosity. I stumbled across 301redirect.it and EasyRedir. 301redirect.it is free, but easyredir.com charges between $10 and $80 per month. I’m sure their service is awesome, but that was a big enough price tag that I started thinking it might be worthwhile to try to compete in the same market. Especially since DNS redirection is so easy.

This is probably the most feature complete software project I’ve ever done. It’s a “full SaaS platform,” meaning it has proper authentication (via Auth0), billing capabilities (via Stripe) , and is totally self-service for customers to consume. It should be a turnkey cash cow.

But like most of my projects, I did effectively zero promotion or sales work, so of course it has zero customers (not just zero paying customers - zero customers period). Before I sunset the project forever, I wanted to do this blog post and put the site up for sale on Flippa.

Before we breakdown the architecture and application, check out the walkthrough video or check out the site for yourself:

DNSReRoute

Architecture

This architecture is not ideal, so before you “Principal Architect” types try to mouth off in the comments, know that I’m quite aware of its design flaws.

At a high level, DNSReRoute is comprised of four components:

RESTful API backend built on Flask running on OpenShift
“Dumb” redirect servers running Flask, in an autoscaling group, behind an ELB in AWS
A MongoDb instance running on mLab with a public endpoint. Both the RESTful API backend and the “dumb” redirect servers read from this DB.
A static, single page app (SPA) using jQuery and hosted on GitHub Pages.

In case that doesn’t make any sense, here’s a sexy diagram to help you understand:

Let’s do an example walkthrough. Let’s pretend we are “BlackLanternStudio” and we sell our product on Etsy, at https://www.etsy.com/shop/BlackLanternStudio. We want to capture more traffic and retain more brand identity, so we want www.BlackLanternStudio.com to redirect to https://www.etsy.com/shop/BlackLanternStudio. To do this, we would browse to www.dnsreroute.xyz, which takes you to the static front-end site. From here we would create an account and then create a “route” to handle our redirect, including what incoming DNS names to expect (inbound routes), and where to redirect that traffic (outbound routes). All of the actions in the UI are making API calls to the Flask backend, which then writes that data to the database. Finally, we would log into our DNS system and create a CNAME record, pointing www.BlackLanternStudio.com to service.dnsreroute.xyz. Now when our customers browse to www.BlackLanternStudio.com, they are actually being sent to DNSReRoute’s redirect hosts. The redirect hosts are running some fairly simple Flask code, basically inspecting the Host header to get the inbound DNS name, looking up the “route” associated with that inbound DNS name in the database, and redirecting the user’s browser to the outbound DNS name.

Originally I was going to host everything on OpenShift to keep things simple, but using custom and un-predictable customer CNAMEs wouldn’t work. In addition, moving the actual redirect work to a separate stack allows the two components to scale independently and prevents bad code/failed deploys to the API layer (where more complex logic resides, more likely to break) from affecting the redirect layer (less complex, less likely to break).

Third Party services

Auth0

Setting up authentication with Auth0 was a little tricky at first, but it has been bulletproof ever since. Auth0 provides a clean Flask example that is easy to integrate with your Flask implementation. If I’m being honest, I think I re-used their authentication wrapper line for line. Most of the time I spent on Auth0 setup didn’t involve any code, but rather all of the ancillary pieces like SSL certificates, setting up the callback URI, registering domains for Cross Origin Requests (CORS) in Auth0’s web UI, etc.

Stripe

Stripe makes handling subscriptions and one-time payments in your web application fun. They abstract all of the difficult parts of payment processing and record keeping, you just have to write your app to consume their APIs for creating new customers and looking up existing subscriptions. I really like Stripe because their ‘sandbox’ or development support is significantly easier than PayPal. With Stripe, you simply change which API client keys you are using, and you can use any fake credit card number to test your app. With PayPal, you have to set up a sandbox, you have to set up sandbox payer/payee accounts, and have to use certain credit card numbers.

TL;DR stripe is awesome for payment processing.

Database design

When I first started this project, it had a much smaller scope. Based solely on storing route information, I decided a NoSQL database like MongoDB would be a perfect fit. Unfortunately, as I added features like user management, organizations, and subscriptions, I had to build relations between my tables/collections. So I’ve effectively created a relational DB on top of a non-relational DB, which is a little embarrassing to say the least.

DNSReRoute uses four collections (like tables, if you’re not familiar):

orgs - comprised of an ID, orgName, and subscription
users - comprised of an ID, userName (used for display purposes), userEmail (used for auth purposes), and orgId as a “foreign key”
routes - comprised of an ID, incomingRoute, outgoingRoute, type (301 vs 302 redirect), and orgId as a “foreign key”
subscriptions - comprised of an ID, subscriptionName (matches the subscription name in Stripe), title (for display purposes), and some boilerplate HTML describing the subscription, so it can be consumed consistently in the UI

For the handful of people out there who can read an ER diagram and appreciate them, here it is:

Static SPA Front-end

The front-end is made with DashGum, a pre-made Dashboard template built using bootstrap that I found on colorlib. All of the logic is written in straight JavaScript and jQuery, nothing fancy like Angular or React. A true front-end developer would probably be appalled with my code, but it works and I was able to knock out a prototype very quickly. The page itself is actually hosted on GitHub pages in a public repo, so the all of the code is available for you to peruse if you’re interested:

Static SPA Repo

API Backend

The API backend is a Flask app running on OpenShift. I chose OpenShift for this project as OpenShift will allow you to keep a certain number of free apps running 24/7, so long as you have a credit card on file. This is in contrast to Heroku (which I also love) that puts your “dynos” to sleep when they become inactive.

The API backend can be hit at https://dnsrerouteprod-dnsreroute.rhcloud.com. You can browse through the Swagger API doc to get a feel for the routes. Note that you won’t be able to send any API calls via the Swagger doc unless the SAML Bearer token set. If you open the Swagger doc from the DNSReRoute dashboard (by clicking on the “API Playground” link in the left sidebar), it will set your SAML Bearer token for you.

As for the code, all of the relevant logic is contained in flaskapp.py:

import os
from datetime import datetime
from urlparse import urlparse
from flask import Flask, request, flash, url_for, redirect, \
     render_template, abort, send_from_directory, jsonify, _request_ctx_stack
import jwt
import requests
import base64
from functools import wraps
from werkzeug.local import LocalProxy
from flask.ext.cors import CORS, cross_origin
import pymongo
from pymongo import MongoClient, IndexModel, ASCENDING, DESCENDING, errors
from bson.json_util import dumps
import json
from bson.objectid import ObjectId
import stripe

env = os.environ
client_id = os.environ['AUTH0_CLIENT_ID']
client_secret = os.environ["AUTH0_CLIENT_SECRET"]
stripe.api_key = os.environ["STRIPE_API_KEY"]

requestorEmail = ""

app = Flask(__name__)
app.config.from_pyfile('flaskapp.cfg')
CORS(app)

# Format error response and append status code.
def handle_error(error, status_code):
  resp = jsonify(error)
  resp.status_code = status_code
  return resp

def requires_auth(f):
  @wraps(f)
  def decorated(*args, **kwargs):
    auth = request.headers.get('Authorization', None)
    if not auth:
      return handle_error({'code': 'authorization_header_missing', 'description': 'Authorization header is expected'}, 401)

    parts = auth.split()

    if parts[0].lower() != 'bearer':
      return handle_error({'code': 'invalid_header', 'description': 'Authorization header must start with Bearer'}, 401)
    elif len(parts) == 1:
      return handle_error({'code': 'invalid_header', 'description': 'Token not found'}, 401)
    elif len(parts) > 2:
      return handle_error({'code': 'invalid_header', 'description': 'Authorization header must be Bearer + \s + token'}, 401)

    # This is the bearer token
    token = parts[1]

    try:
        payload = jwt.decode(
            token,
            base64.b64decode(client_secret.replace("_","/").replace("-","+")),
            audience=client_id
        )
    except jwt.ExpiredSignature:
        return handle_error({'code': 'token_expired', 'description': 'token is expired'}, 401)
    except jwt.InvalidAudienceError:
        return handle_error({'code': 'invalid_audience', 'description': 'incorrect audience, expected: ' + client_id}, 401)
    except jwt.DecodeError:
        return handle_error({'code': 'token_invalid_signature', 'description': 'token signature is invalid'}, 401)
    except Exception:
        return handle_error({'code': 'invalid_header', 'description':'Unable to parse authentication token.'}, 400)

    endpoint = "https://alexdglover.auth0.com/tokeninfo"
    headers = {"Authorization":"Bearer " + token}
    data = {"id_token": token}
    global requestorEmail
    try:
        requestorEmail = requests.post(endpoint,data=data,headers=headers).json()['email']
    except Exception as e:
        return handle_error({'code': 'failed_user_lookup', 'description': 'Unable to look up user with that token'})

    _request_ctx_stack.top.current_user = user = payload
    return f(*args, **kwargs)

  return decorated

def connect():
    connection = MongoClient(os.environ['OPENSHIFT_MONGODB_DB_URL'],int(os.environ['OPENSHIFT_MONGODB_DB_PORT']))
    handle = connection["dnsreroute"]
    handle.authenticate(os.environ['OPENSHIFT_MONGODB_DB_USERNAME'],os.environ['OPENSHIFT_MONGODB_DB_PASSWORD'])
    return handle

def initializeDb():
    handle.routes.create_index( [ ("incomingRoute", ASCENDING)], unique=True )
    handle.users.create_index( [ ("userEmail", ASCENDING)], unique=True )

handle = connect()

initializeDb()

# Sets response headers for all requests received. This is needed to allow
# pre-flight OPTIONS requests to get the information they need to do PUTs
# and DELETEs
@app.after_request
def after_request(response):
  response.headers['Access-Control-Allow-Origin'] = request.headers.get('Origin','*')
  response.headers.add('Access-Control-Allow-Headers', 'Content-Type,Authorization')
  response.headers.add('Access-Control-Allow-Methods', 'PUT,POST,OPTIONS,DELETE,GET')
  return response

# Controllers API
@app.route("/")
def home():
    host = request.headers['Host']
    host = host.split(':')[0]
    if host != 'service.dnsreroute.xyz':
        route = handle.routes.find_one({'incomingRoute': host})
        if route:
            outgoingRoute = route['outgoingRoute']
            if route['type'] == "301":
                return redirect(outgoingRoute, 301)
            elif route['type'] == "302":
                return redirect(outgoingRoute)
            else:
                return '{"message": "Error - not able to determine redirect type"}'
        else:
            return '{"message": "Could not find a matching route"}', 404

    else:
      return "The Host header is {hostHeader}.This is the unsecured home page".format(hostHeader=request.headers['Host'])

@app.route("/ping",host="*")
def ping():
    return "All good. You don't need to be authenticated to call this"

@app.route("/secured/ping")
@requires_auth
def securedPing():
    return "All good. You only get this message if you're authenticated"

###################################
###   Route URIs
###################################

@app.route("/routes/byUserEmail/<userEmail>", methods=['GET'])
@requires_auth
def getRoutesByUserEmail(userEmail):
    orgId = handle.users.find_one({"userEmail": userEmail})['orgId']
    print 'orgId is {orgId}'.format(orgId=orgId)
    routes = handle.routes.find({"orgId": ObjectId(orgId)})
    if routes:
        return dumps(routes)
    else:
        message = {"message": "No routes found with that orgId"}
        return jsonify(message), 404

@app.route("/routes/byOrg/<orgId>", methods=['GET'])
@requires_auth
def getRoutesByOrgId(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    routes = handle.routes.find({"orgId": orgId})
    if routes:
        return dumps(routes)
    else:
        message = {"message": "No routes found with that orgId"}
        return jsonify(message), 404


@app.route("/routes", methods=['POST'])
@requires_auth
def addRoute():
    if isAuthorized(requestorEmail, 'addRoute'):
        try:
            handle.routes.insert({"orgId": ObjectId(request.values['orgId']), "type": request.values['type'],
                "incomingRoute": request.values['incomingRoute'], "outgoingRoute": request.values['outgoingRoute']})
            return '{"message": "successfully added route"}'
        except pymongo.errors.DuplicateKeyError:
            errorDict = {"message": "Failed to add route - that incoming DNS name is already in use"}
            return jsonify(errorDict), 400
        except Exception as e:
            errorDict = {"message": "Failed to add route. Error message: {error}".format(error=e)}
            return jsonify(errorDict), 400
    else:
        # Not authorized
        errorDict = {"message": "You are not authorized to add another route!"}
        return jsonify(errorDict), 403

@app.route("/routes/<incomingRoute>", methods=['DELETE'])
@requires_auth
def deleteRoute(incomingRoute):
    # Validate incomingRoute is valid first
    route = handle.routes.find_one({"incomingRoute": incomingRoute})
    if route:
        if isAuthorized(requestorEmail, 'deleteRoute', incomingRoute):
            try:
                result = handle.routes.remove({"incomingRoute": incomingRoute})
                return '{"message": "Successfully deleted route"}'
            except Exception as e:
                errorDict = {"message": "Failed to delete route. Error message: {error}".format(error=e)}
                return jsonify(errorDict), 400
        else:
            # Not authorized
            errorDict = {"message": "You are not authorized to delete that route!"}
            return jsonify(errorDict), 403
    # If the route targeted for deletion wasn't found, return a 200 with explanation
    else:
        message = {"message": "Route doesn't exist, but that's OK! HTTP DELETE is an idempotent operation dude"}
        return jsonify(message)

###################################
###   End of Route URIs
###################################

###################################
###   User URIs
###################################

@app.route("/users")
@requires_auth
def getUsers():
    actor = handle.users.find_one({"userEmail": requestorEmail})
    users = handle.users.find({"orgId": ObjectId(actor['orgId'])})
    if users:
        return dumps(users)
    else:
        message = {"message": "No users found"}
        return jsonify(message), 404

@app.route("/users/byOrg/<orgId>")
@requires_auth
def getUsersByOrg(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    users = handle.users.find({"orgId": orgId})
    if users:
        return dumps(users)
    else:
        message = {"message": "No users found with that org ID"}
        return jsonify(message), 404

@app.route("/users/<userEmail>", methods=['GET'])
@requires_auth
def getUserByEmail(userEmail):
    user = handle.users.find_one({"userEmail": userEmail})
    if user:
        user['_id'] = str(user['_id'])
        user['orgId'] = str(user['orgId'])
        return dumps(user)
    else:
        message = {"message": "No user found with that email address"}
        return jsonify(message), 404

@app.route("/users/<userEmail>", methods=['PUT'])
@requires_auth
def updateUser(userEmail):
    if requestorEmail == userEmail:
        user = handle.users.find_one({"userEmail": userEmail})
        if user:
            result = handle.users.update_one( { "userEmail": userEmail },
                {
                  "$set": {
                    # "userEmail": userEmail,
                    "userName": request.values['userName'],
                    "orgId": ObjectId(request.values['orgId'])
                  }
                }
            )
            message = {"message": "User updated successfully"}
            return jsonify(message)
        else:
            message = {"message": "No user found with that email address"}
            return jsonify(message), 404
    else:
        message = {"message": "You are not authorized to update that user. You may only update your own user information"}
        return jsonify(message), 403

@app.route("/users", methods=['POST'])
@requires_auth
def addUser():
    if isAuthorized(requestorEmail, 'addUser', request.values['orgId']):
        try:
            handle.users.insert({"userEmail":request.values['userEmail'], "userName":request.values['userName'], "orgId": ObjectId(request.values['orgId']) })
            message = {"message": "Successfully added user"}
            return jsonify(message)
        except pymongo.errors.DuplicateKeyError:
            errorDict = {"message": "User with that email already exists"}
            return jsonify(errorDict), 409
        except Exception as e:
            errorDict = {"message": "Failed to add user. Error message: {error}".format(error=e)}
            return jsonify(errorDict), 400
    else:
        # Not authorized
        errorDict = {"message": "You are not authorized to add another user!"}
        return jsonify(errorDict), 403

@app.route("/users/register", methods=['POST'])
@requires_auth
def registerNewUser():
    try:
        orgId = handle.orgs.insert({"orgName": request.values['userEmail'], "subscription": "freeTier"})
        handle.users.insert({"userEmail":request.values['userEmail'], "userName":request.values['userName'], "orgId": ObjectId(orgId) })
        message = {"message": "Successfully registered user"}
        return jsonify(message)
    except pymongo.errors.DuplicateKeyError:
        errorDict = {"message": "User with that email already exists"}
        return jsonify(errorDict), 409
    except Exception as e:
        errorDict = {"message": "Failed to add user. Error message: {error}".format(error=e)}
        return jsonify(errorDict), 400

@app.route("/users/<userEmail>", methods=['DELETE'])
@requires_auth
def deleteUser(userEmail):
    targetUser = handle.users.find_one({"userEmail": userEmail})
    if targetUser:
        if isAuthorized(requestorEmail, 'deleteUser', userEmail):
            try:
                result = handle.users.remove({"userEmail": userEmail})
                message = {"message": "Successfully deleted user"}
                return jsonify(message)
            except Exception as e:
                errorDict = {"message": "Failed to delete user. Error message: {error}".format(error=e)}
                return jsonify(errorDict), 400
        else:
            # Not authorized
            errorDict = {"message": "You are not authorized to delete that user!"}
            return jsonify(errorDict), 403
    # If the user targeted for deletion wasn't found, return a 200 with explanation
    else:
        message = {"message": "User doesn't exist, but that's OK! HTTP DELETE is an idempotent operation dude"}
        return jsonify(message)

###################################
###   End of User URIs
###################################

###################################
###   Org URIs
###################################
@app.route("/orgs", methods=['POST'])
@requires_auth
def addOrg():
    try:
        orgId = handle.orgs.insert({"orgName": request.values['userEmail'], "subscription": "freeTier"})
        return '{"message": "Successfully added org"}'
    except pymongo.errors.DuplicateKeyError:
        errorDict = {"message": "Org already exists"}
        return jsonify(errorDict), 400
    except Exception as e:
        errorDict = {"message": "Failed to add org. Error message: {error}".format(error=e)}
        return jsonify(errorDict), 400

@app.route("/orgs/<orgId>", methods=['PUT'])
@requires_auth
def updateOrg(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    try:
        org = handle.orgs.find_one( {"_id": orgId} )

        if org:
            result = handle.orgs.update_one( {"_id": orgId},
                {
                  "$set": {
                    "orgName": request.values['orgName']
                  }
                }
            )
            message = {"message": "Org updated successfully"}
            return jsonify(message)
        else:
            message = {"message": "No org with that id"}
            return jsonify(message), 404
    except Exception as e:
        errorDict = {"message": "Failed to update org. Error message: {error}".format(error=e)}
        return jsonify(errorDict), 400

@app.route("/orgs/<orgId>", methods=['GET'])
@requires_auth
def getOrg(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    org = handle.orgs.find_one({"_id": orgId})
    if org:
        org['_id'] = str(org['_id'])
        return dumps(org)
    else:
        message = {"message": "No org found with that orgId"}
        return jsonify(message), 404

@app.route("/orgs/<orgId>", methods=['DELETE'])
@requires_auth
def deleteOrg(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    org = handle.orgs.find_one({"_id": orgId})

    if isAuthorized(requestorEmail, 'deleteOrg', orgId):
        # Cancel Stripe subscription
        if 'subscriptionId' in org:
            try:
                subscriptionId = org['subscriptionId']
                subscription = stripe.Subscription.retrieve(subscriptionId)
                subscription.delete()
            except Exception as e:
                errorDict = {"message": "Error occurred while updating existing subscription. Error message: {error}".format(error=e)}
                return jsonify(errorDict), 400

        # Delete all associated user accounts
        handle.users.remove({"orgId": orgId})

        # Delete all associated routes
        handle.routes.remove({"orgId": orgId})

        try:
            result = handle.orgs.remove({"_id": orgId})
            return '{"message": "Successfully deleted org"}'
        except Exception as e:
            errorDict = {"message": "Failed to delete org. Error message: {error}".format(error=e)}
            return jsonify(errorDict), 400
    else:
        # Not authorized
        errorDict = {"message": "You are not authorized to delete that org!"}
        return jsonify(errorDict), 403


@app.route("/orgs/<orgId>/subscription", methods=['PUT'])
@requires_auth
def addSubscriptionToOrg(orgId):
    orgId = validateObjectId(orgId)

    if not orgId:
        message = {"message": "Invalid org ID, uanble to convert to ObjectId. Must be a 12-byte input or a 24-character hex string"}
        return jsonify(message), 400

    # Check org for existing customer ID
    try:
        org = handle.orgs.find_one( {"_id": orgId} )
        if org:
            print "org is:"
            print org
            if 'subscriptionId' in org:
                try:
                    subscriptionId = org['subscriptionId']
                    subscription = stripe.Subscription.retrieve(subscriptionId)
                    subscription.plan = request.values['subscription']
                    subscription.save()
                except Exception as e:
                    errorDict = {"message": "Error occurred while updating existing subscription. Error message: {error}".format(error=e)}
                    return jsonify(errorDict), 400

                result = handle.orgs.update_one( {"_id": orgId},
                    {
                      "$set": {
                        "subscription": request.values['subscription']
                      }
                    }
                )
                message = {"message": "Org updated with subscription successfully"}
                return jsonify(message)
            else:
                try:
                    stripeCustomer = stripe.Customer.create(
                      source=request.values['tokenId'], # obtained from Stripe.js
                      plan=request.values['subscription'],
                      email=request.values['userEmail']
                    )
                    print "stripeCustomer is:"
                    print stripeCustomer
                    subscriptionId = stripeCustomer.subscriptions.data[0].id
                except Exception as e:
                    errorDict = {"message": "Error occurred while creating new user and subscription. Error message: {error}".format(error=e)}
                    return jsonify(errorDict), 400
                result = handle.orgs.update_one( {"_id": orgId},
                    {
                      "$set": {
                        "subscription": request.values['subscription'],
                        "subscriptionId": subscriptionId
                      }
                    }
                )
                message = {"message": "Org updated with subscription successfully"}
                return jsonify(message)
        else:
            message = {"message": "No org with that id"}
            return jsonify(message), 404
    except Exception as e:
        errorDict = {"message": "Failed to update org. Error message: {error}".format(error=e)}
        return jsonify(errorDict), 400

###################################
###   End of Org URIs
###################################



###################################
###   Subscription URIs
###################################

@app.route("/subscriptions/<subscriptionName>", methods=['GET'])
@requires_auth
def getSubscription(subscriptionName):
    subscription = handle.subscriptions.find_one({"subscriptionName": subscriptionName})
    if subscription:
        subscription['_id'] = str(subscription['_id'])
        return dumps(subscription)
    else:
        message = {"message": "No subscription found with that subscriptionName"}
        return jsonify(message), 404

###################################
###   End of Subscription URIs
###################################

###################################
###   Non-URI Functions
###################################

def validateObjectId(objectId):
    try:
        objectId = ObjectId(objectId)
        return objectId
    except:
        return None

def isAuthorized(actorEmail, action, target=None):
    actor = handle.users.find_one({"userEmail": actorEmail})
    org = handle.orgs.find_one({"_id": ObjectId(actor['orgId'])})
    print org
    maxRoutes = { 'freeTier': 1, 'developerTier': 10, 'enterpriseTier': 100 }
    if action == 'addUser':
        if (org['subscription'] == 'developerTier') or (org['subscription'] == 'enterpriseTier'):
            print 'Subscription is not freeTier, requesting user is authorized to add that user. Checking target org'
            if ObjectId(target) == actor['orgId']:
                print 'Target org matches requesting users org, request is authorized'
                return True
            else:
                return True
        else:
            print 'Subscription is freeTier or some unhandled value, requesting user is NOT authorized to add target user'
            return False

    elif action == 'deleteUser':
        targetUser = handle.users.find_one({"userEmail": target})
        if (org['subscription'] == 'developerTier') or (org['subscription'] == 'enterpriseTier'):
            if actorEmail != target:
                if actor['orgId'] == targetUser['orgId']:
                    print 'Subscription is not freeTier, the user is not deleting themself, and this user is part of the same org. Requesting user is authorized to delete target user'
                    return True
                else:
                    print 'Subscription is not freeTier, the user is not deleting themself, but the user is NOT part of the same org. Requesting user is NOT authorized to delete target user'
                    return False
            else:
                print 'Subscription is not freeTier, but the user is attempting to delete themself. Requesting user is NOT authorized to delete target user'
                return False
        else:
            print 'Subscription is freeTier or some unhandled value, requesting user is NOT authorized to delete target user'
            return False

    elif action == 'addRoute':
        routeCount = handle.routes.find({"orgId": org['_id']}).count()
        if routeCount < maxRoutes[org['subscription']]:
            print 'Current routeCount is less than maxRoutes, requesting user is authorized to create target route'
            return True
        else:
            print 'Current routeCount is equal to or greater than maxRoutes, requesting user is NOT authorized to create target route'
            return False
    elif action == 'deleteRoute':
        targetRoute = handle.routes.find_one({"incomingRoute": target})
        if org['_id'] == targetRoute['orgId']:
            print 'User is trying to delete a route that is owned by their org, requesting user is authorized to delete target route'
            return True
        else:
            print 'User is trying to delete a route that is NOT owned by their org, requesting user is NOT authorized to delete target route'
            return False
    elif action == 'deleteOrg':
        targetOrg = handle.orgs.find_one({"_id": ObjectId(target)})
        if org['_id'] == targetOrg['_id']:
            print 'User is trying to delete their own org, requesting user is authorized to delete target org'
            return True
        else:
            print 'User is trying to delete some other org, requesting user is NOT authorized to delete target org'
            return False
    else:
        print 'Unable to determine action, requesting user is NOT authorized'
        return False


###################################
###   End of Non-URI Functions
###################################


if __name__ == '__main__':
    app.run(app.config['IP'], app.config['PORT'])

If you want to see the rest of the files or fork my repo, you can find it on GitHub:

Flask Backend Repo

Redirecting Backend

The “dumb” redirect hosts are running a stripped down, simpler version of the Flask app used by the API backend. The redirect hosts are part of an auto-scaling group with an Elastic Load Balancer distributing requests. The configuration of the hosts is all done through a userdata script, which makes auto-scaling simple and removes the need for configuration management tools.

In my haste, I lumped together all of the code for the redirecting app and the CloudFormation code into one repo. Check out the repo here:

Redirecting Backend Repo

The Flask app for handling redirects is pretty straightforward. All of the relevant logic is written in the redirectorApp.py file:

import os
from flask import Flask, request, flash, url_for, redirect, \
     render_template, abort, send_from_directory, jsonify, _request_ctx_stack
import requests
from werkzeug.local import LocalProxy
from flask.ext.cors import CORS, cross_origin
import pymongo
from pymongo import MongoClient, IndexModel, ASCENDING, DESCENDING, errors


app = Flask(__name__)
app.config.from_pyfile('flaskapp.cfg')
CORS(app)


def connect():
    print os.environ['OPENSHIFT_MONGODB_DB_URL']
    print int(os.environ['OPENSHIFT_MONGODB_DB_PORT'])
    print os.environ['OPENSHIFT_MONGODB_DB_USERNAME']
    print os.environ['OPENSHIFT_MONGODB_DB_PASSWORD']
    connection = MongoClient(os.environ['OPENSHIFT_MONGODB_DB_URL'],int(os.environ['OPENSHIFT_MONGODB_DB_PORT']))
    handle = connection["dnsreroute"]
    handle.authenticate(os.environ['OPENSHIFT_MONGODB_DB_USERNAME'],os.environ['OPENSHIFT_MONGODB_DB_PASSWORD'])
    return handle

handle = connect()

# Controllers API
@app.route("/")
def home():
    host = request.headers['Host']
    host = host.split(':')[0]
    if host != 'service.dnsreroute.xyz':
        route = handle.routes.find_one({'incomingRoute': host})
        if route:
            outgoingRoute = route['outgoingRoute']
            if route['type'] == "301":
                return redirect(outgoingRoute, 301)
            elif route['type'] == "302":
                return redirect(outgoingRoute)
            else:
                return '{"message": "Error - not able to determine redirect type"}'
        else:
            return '{"message": "Could not find a matching route"}', 404

    else:
      return "The Host header is {hostHeader}.This is the unsecured home page".format(hostHeader=request.headers['Host'])

# Controllers API
@app.route("/health")
def healthCheck():
    return '{"message": "System OK!"}'


if __name__ == '__main__':
    app.run(app.config['IP'], app.config['PORT'])

One implementation note - I was originally using the eventlet Gunicorn worker type, but as I was writing this blog post and testing things were still working I discovered that my instances were no longer bootstrapping successfully. Turns out it was some gunicorn/eventlet/monotonic error throwing a RuntimeError of no suitable implementation for this system. Rather than go down that rabbit hole, I just switched the worker type to gevent and moved on.

Conclusion

I’ll be leaving DNSReRoute running at least until July 2017, after which my 12 month free trial of AWS expires. If I haven’t stirred up any interest or customers by then, I’ll terminate the redirecting hosts, but keep the API backend and front-end running for reference purposes.

I had a lot of fun writing this app, and I’m proud of it in spite of its flaws and lack of use. I hope you were able to cherry pick some code or ideas from this post, or some inspiration if nothing else.

Thanks for stopping by.

BASH Hacks as First Class Commands

2017-05-11T00:00:00+00:00

In my new job at Zendesk, I deal with a lot of Ruby projects that leverage bundler for dependency management. Specifically, right now I’m working with SparkleFormation, stack_master, and aws-vault. With that toolset, each command starts with a staggering 43 static characters that don’t change between commands, like this:

aws-vault exec <AWS profile> -- bundle exec stack_master apply \
<region or alias> <stack name> -c <stack_master config file>

That’s way too long to type out every time I need to act on a stack. Initially I tried to solve this by setting an alias in my BASH profile to abstract the stack_master command:

alias sm=”bundle exec stack_master”

One problem with this approach - if you use a command or binary that spawns a new shell (like aws-vault does), your bash_profile is ignored and your aliases don’t work, defeating the entire purpose.

Fortunately this is an easy problem to solve. Because the default PATH is the same for all shells, we can recreate our alias as a script and place it in /usr/local/bin (or any other directory in the default PATH). We can also leverage BASH’s $@ variable, which captures all positional arguments that come after a script is invoked.

#!/bin/bash

bundle exec stack_master “$@”

Once I created /usr/local/bin/sm and populated it with those lines, executing sm behaved exactly as the original alias, but was also compatible with aws-vault.

For those of you using the same toolset of aws-vault, stack_master, and SparkleFormation, I further condensed my command. Note that this only works if your aws-vault profile, region alias, and stack_master config files share the same name.

#!/bin/bash

# Example command for reference
# aws-vault exec dev -- bundle exec stack_master apply dev stack_name -c dev.yml

aws-vault exec $1 -- bundle exec stack_master $2 $1 $3 -c $1.yml

I put the above code in /usr/local/bin/zsm, so now my stack commands are clean and short:

zsm dev apply my_stack

As always, I hope you found this useful.

Dennis Nedry Failed Login

2017-03-11T21:22:08+00:00

This one’s a short read, as most of the code and the concept was adopted from the jQuery Raptorize plugin. Basically, I was admiring the Raptorize plug-in and thought how funny it would be to have Dennis Nedry pop up and deliver his famous line, “Ah ah ah, you didn’t say the magic word!” whenever you had a failed login. Thanks to the jQuery Raptorize plugin, the majority of the work was done. I just needed a picture of Dennis Nedry, an mp3 file for the audio, and tweak the code slightly. Then I put together a fake login page to demo the whole thing.

Demo GitHub Repo

This was a quick 1-2 hour kind of project that I did a long time ago and just never wrote a blog post for it. Hoping you got a quick laugh out of this post, let me know if any of you actually deploy it somewhere!

The Move to Jekyll

2017-03-04T15:03:23+00:00

WordPress has served me well over the years, but it’s time for me to move on. My blog first started with Wordpress.com hosting, but I didn’t like that I was hosting WP’s ads, a custom domain was an up-charge, and they limit which themes and plugins I could use. So I migrated to a self-hosted WordPress instance, allowing me to serve Google ads, add membership functionality, and allow people to subscribe to my site by email.

I’ve had some issues with 1and1 hosting over the years (downtime, charges for PHP support, non-refundable contracts), and I’ve come to realize I don’t need most of the plugins. I don’t make enough money to justify the ads, and a lot of the site’s visitors are friends and family anyway. I only have a few members/subscribers, so I don’t really need membership functionality either. On top of that, 1and1 isn’t free - I’m literally paying for a hosting service I don’t like, to host functionality I don’t need.

With all of that in mind, blogging with Jekyll and hosting on GitHub seemed like a no-brainer. Hosting is free, it’s faster and more secure than WP, and I get to enjoy hacking on a new tool set. But before I made the switch, I wanted to make sure Jekyll could do everything I wanted with a blogging platform.

TL;DR - I’m ditching WordPress hosted on 1and1, and I’m moving to Jekyll hosted on GitHub.

Reproducing features in Jekyll

Out of the box, Jekyll is a great blogging platform and meets the majority of my use cases. That said, there a few features of my WordPress blog that Jekyll doesn’t include, and I didn’t want to lose as part of the transition.

Featured image functionality - basically, I can set an image for a post, and it shows up in any list of posts, as well as the first image you see when you open a specific blog post.
JetPack site statistics - Part of the satisfaction of blogging is seeing how much traffic you get. WordPress does a great job with their JetPack plugin, which allows you to get fairly substantial analytics in a simple and consumable dashboard.
Social media integration - when I publish a post, WordPress shares it to Twitter, Facebook, and LinkedIn automatically. There are also sharing buttons appended to every post.
Retaining permalinks for SEO and backlinks - SEO is not very important for my humble blog, but I hate when I try to browse to some personal blog only to get a 404 or weird redirect. With that in mind, Jekyll needs to support my existing permalink structure (e.g. alexdglover.com/my-post-title)
Search - users should be able to find content based on query text.

The rest of this post will cover how I modified Jekyll and the Minimal Mistakes theme for my blog.

Featured Image Functionality

Fortunately, implementing the “featured image” was fairly easy with the Minimal Mistakes theme. The theme itself is supposed to offer it out-of-the-box, but either I’m doing something wrong or misunderstanding the theme documentation. In theory, you only need to set “teaser” property in the header section of your posts’ front matter, like this:

---
title:  "Hello world"
date:   2016-11-17 16:16:01 -0600
categories: jekyll category1 category2
header:
  teaser: https://i0.wp.com/alexdglover.com/wp-content/uploads/2017/01/anim-3.gif
---

And here's the body of your blog post.

For some reason, that didn't work for me using the stock home page. After doing some digging, I narrowed the so-called issue to a "grid" conditional in _includes/archive-single.html:

{% if include.type == "grid" and teaser %}
  <div class="archive__item-teaser">
    <img src=
      {% if teaser contains "://" %}
        "{{ teaser }}"
      {% else %}
        "{{ teaser | prepend: "/images/" | prepend: base_path }}"
      {% endif %}
      alt="">
  </div>
{% endif %}

Well, I couldn't figure out where the include.type was supposed to be set, but who cares! The best thing about Jekyll is you have total control and it's easy to modify. I have no plans to use this "grid" functionality, so I just removed it from the conditional. Now the teaser is displayed on the home page (and in any page using the archive layout).

Site Analytics and Stats

Jekyll supports Google Analytics, injecting your tracking ID into every page on your behalf. Google Analytics is a little overkill for a personal blog and I personally dislike the dashboard/interface, but there aren’t that many site analytics platforms out there that are free and feature-rich. So I’m going down the path of least resistance on this one. I just need to take my existing GA tracking ID, and put in the _config.yml file:

# Analytics
analytics:
  provider       : "google"
  google:
    tracking_id  : "ga-my-tracking-id-1234556789"

I’ve seen plenty of other Jekyll users create a simple integration to social media sites using the built-in RSS feed from Jekyll and IFTTT, so I won’t beat that dead horse. The most interesting implementation I’ve seen done was by Eduardo Boucas, so I’ll direct you to his site for an example.

Retaining Permalinks

Fortunately, when I set up my self-hosted WordPress site I had the foresight to choose a sane and practical permalink structure. All of my permalinks are just the post title, but URL encoded (99% of the time this just means spaces are replaced by hyphens, and all words are downcased). I can maintain all of my permalinks and not lose any SEO “juice” by using that setting the same permalink pattern in the _config.yml of my new Jekyll blog:

# Outputting
permalink: /:title/

Easy.

Search

Search functionality was an interesting problem, but I knew right away how I wanted to solve it. In short, because Jekyll is a static site generator, there is no server-side code that can process search queries. It all needs to be handled in HTML and JavaScript.

This is a trivial limitation, considering how small my site is and how “cheap” it is to send a few extra kilobytes of HTML text to the end user. With that in mind, a search solution is fairly easy. In my initial implementation, I set up a separate search page (rather than try to do an _includes/search.html that I could embed in other pages or sidebars). The search page is populated with an HTML input field, followed by every single post in the blog wrapped in div tags with style="display: none;" . At this point, the DOM of the page becomes our search “database,” and we just need to write some JavaScript to un-hide our divs based on the search query. This gives us the power to search not only post titles, but also the actual content of the post bodies.

---
title: Search
permalink: search.html
---

<script>
function getSearchResults(val) {
  console.log("hiding all elements");
  $( ".result" ).hide();
  $( ".postContent" ).hide();
  $( ".searchResultSnippet " ).hide();

  if(val.length > 2){
    // Downcase query val for case-insensitive search
    val = val.toLowerCase();

    // Show any results based on post title matching query
    $( "div[id*='" + val + "' i]" ).filter( ".result").show();

    // Show any results where query matches part of the post content
    $( "div:contains('" + val + "')" ).filter( ".result").show();

    // For each result based on post content, extract the surrounding text
    // and display it in the search results, bolding the search term
   $( "div:contains('" + val + "')" ).filter( ".result").each( function( index ) {
      var postText = $( this ).find( ".postContent" ).text();
      var resultPosition = postText.indexOf(val);
      if(resultPosition > -1){
        var snippet = "..." + postText.substring( resultPosition-30, resultPosition+30 ) + "...";
        snippet = snippet.replace(val, "<strong>" + val + "</strong>");
        $( this ).find( ".searchResultSnippet" ).show().html(snippet);
      }
    });
  } // End of input length check

} // End of getSearchResults
</script>

<input type="text" onkeyup="getSearchResults(this.value)">

<div id="results">
  {% for post in site.posts %}
  <div id="{{ post.title }}" class="result" style="display: none;">
    <h3><a href="{{ post.url }}">{{ post.title }}</a></h3>
    {% if post.header.teaser %}
    <a href="{{ post.url }}"><img src="{{ post.header.teaser }}" style="width: 150px;" class="align-left" /></a>
    {% endif %}
    <p><small><strong>{{ post.date | date: "%B %e, %Y" }}</strong> - <strong>Categories</strong> {{ post.categories | join:',' }} </small></p>
    <div class="postContent" style="display:none;">{{ post.content }}</div>
    <div id="searchResult{{ post.url }}" class="searchResultSnippet" style="display:none;"></div>
  </div>
  {% endfor %}
</div>

And here’s the end result:

It even looks good on mobile-friendly display port sizes:

Conclusion

Well I’m sold! After testing my minor customizations, I’m ready to make the switch to GitHub pages using Jekyll and the Minimal Mistakes theme. I’ll be sure to post again to describe any lessons learned from the actual migration, as well as any future customizations I make. If any of you have Jekyll tips and tricks to share, send ‘em my way!

Removing Secrets from GitHub

2017-02-16T18:00:53+00:00

I did it. The one universal rule when it comes to source control/Git, and I broke it. I committed my OAuth key for my YouPassButter Slack app to GitHub. Thankfully it was only a personal project, and not anything more important than that. Either way, I got a little spooked and wanted to go through the process of removing sensitive data from GitHub as a learning opportunity. Before we get started on the walk-through, there are two things I’d like to mention upfront:

Because of the distributed nature of Git, if someone cloned or forked your repo while it contained sensitive data, that data is compromised. There is no way to delete your data out of someone else’s repo
I will be following GitHub’s published process using the ‘git filter-branch’ command. I’m mostly documenting this for my own future reference, and as a quasi-punishment for myself to emphasize the importance of not committing keys in the future (I could write “Don’t commit keys to GitHub” 1000 times on a blackboard, but this seems more productive)

Process

If you don’t already have the repo cloned locally, do that first, then ‘cd’ into your repo directory.

In my case, the sensitive data was in the ‘app.rb’ file in the root of my project repo. Before you execute this next command, be aware that it will delete the local copy of the file along with updating all of the commits in your commit history. You probably want to save a copy of your file somewhere before running this. With that in mind, let’s execute the ‘git filter-branch’ command and look at the output:

[alexdglover@host:~/Documents/workspace/rick-and-morty-bot] $ git filter-branch --force --index-filter \
> 'git rm --cached --ignore-unmatch app.rb' \
> --prune-empty --tag-name-filter cat -- --all
Rewrite c6a89ccf770fe1745259af4dd5c1de740a8c2a1e (1/75) (0 seconds passed, remaining 0 predicted)    rm 'app.rb'
Rewrite 17e90caa961fa7b801701abfbae9ce0c34a2a9d1 (2/75) (0 seconds passed, remaining 0 predicted)    rm 'app.rb'
Rewrite ec633eb63f1c7eead43157e100006029cf8029a6 (3/75) (0 seconds passed, remaining 0 predicted)    rm 'app.rb'
...
Rewrite c4d38d45a710b7e87c2d75820fe65d204f9014e1 (39/75) (1 seconds passed, remaining 0 predicted)    rm 'app.rb'
Rewrite d2b25cdd3e6190b2d59c1239d5955245c76a173b (39/75) (1 seconds passed, remaining 0 predicted)    rm 'app.rb'
Rewrite c4570086709c08364d769bcb1818c0a2631ea111 (39/75) (1 seconds passed, remaining 0 predicted)    rm 'app.rb'

Ref 'refs/heads/master' was rewritten
Ref 'refs/remotes/heroku/master' was rewritten
WARNING: Ref 'refs/remotes/heroku/master' is unchanged
Ref 'refs/remotes/origin/master' was rewritten
[alexdglover@host:~/Documents/workspace/rick-and-morty-bot] $

The command is going through each commit in the commit history and deleting the file containing the sensitive data. So far, so good.

Typically, in this step you’d create a .gitignore file, add the offending file containing sensitive data to the .gitignore, and then re-commit everything. In my case, I can’t add app.rb to .gitignore because it’s the bulk of my code. Instead, let’s try removing the credentials manually (and replacing it with environment variable references), saving that file into our repo, re-committing app.rb, and see what happens.

[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot]$ vim app.rb # I re-created the app.rb file without the sensitive data
[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot]$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

  app.rb

nothing added to commit but untracked files present (use "git add" to track)
[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot]$ git add app.rb
[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot]$ git commit -m 'adds app.rb back into repo after removing secrets'
[master 346e3e5] adds app.rb back into repo after removing secrets
 1 file changed, 202 insertions(+)
 create mode 100644 app.rb
[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot]$ git push

You can see that Git thinks app.rb is a completely new file, and is unaware that it ever existed. Another good sign. My ‘git push’ had a minor merge conflict with my GitHub upstream repo, but was trivial. Now I’ve strayed slightly from GitHub’s instructions, let’s move on to the last step and see if this works.

Finally, we need to execute ‘git push origin –force –all’ to overwrite our GitHub repository. Let’s give it a try:
```
[alexdglover@mobile1:~/Documents/workspace/rick-and-morty-bot] $ git push origin --force --all
Everything up-to-date
```
Well… that was a little anti-climactic. I’m assuming my earlier push already overwrote GitHub. Let’s take a look in GitHub’s web UI and see if we can find any trace of our sensitive data.

Excellent! I don’t see the Slack client ID and secret in the code any more, and the commit history for that file only shows our most recent commit!
My repo is very simple and didn’t use any other tags or branches. If you committed a secret/key/password to your repo and you do use tags, you’ll also need to execute ‘git push origin –force –tags’ to overwrite all of the tagged commits.

Post Cleanup Process

If you’re working with other developers, be sure to ask them to rebase (not merge) their local repos after you’ve cleaned up the sensitive data, otherwise they may re-introduce bad code or commit history containing the bad code. If you haven’t done so already, this would also be a good time to change any credentials that you think may have been compromised. In my case, I simply needed to generate a new OAuth client secret for my Slack app.

I hope this was a helpful demonstration, and remember…

DON'T PUT CREDENTIALS INTO SOURCE CONTROL

YouPassButter Slack Bot, Part 2

2017-02-04T16:00:23+00:00

TL;DR Section

Just looking for the app? Just click the "Add to Slack" button to install the YouPassButter bot

Prefer to just go straight to the source? Check out the code on GitHub.

Where we left off

If you haven't ready it already, check out Part 1 to get the context behind the app and the initial implementation.

In this post, we'll

re-structure our Slash Commands and routes to get more functionality
handle our second requirement, namely generating dynamic memes and POSTing them back to Slack
package our Slash Commands as a Slack app, including the OAuth configuration needed for the "Add to Slack" button

Re-structuring our commands

In Part 1, we had a one-to-one mapping from our Slash Commands to our API endpoints. This made our app very clean and simple, but has one serious drawback; Slack only allows up to 25 slash commands per 'app.' We haven't even implemented our meme generation feature yet, so this is a dealbreaker for us.

Let's consolidate our image-specific Slash commands into one image-fetching Slash command, and we'll pass a second argument to the command to specify which image we want to return.

Since we're consolidating the Slash commands, we'll also need to consolidate our API endpoints to match. Fortunately, this is very easy. All of our Sinatra routes are basically key-value pairs, where the key is the route (e.g. /you-pass-butter) and the value is the URL to the image (e.g. https://media.giphy.com/whatever.gif). Because of this, we can consolidate all of our routes into a Hash and write a single route to dynamically fetch and return the correct image.

COMMAND_IMAGE_MAPPING = {
  "nobody-exists-on-purpose" => "http://cdn.smosh.com/sites/default/files/2015/12/rickmorty15.jpg",
  "wriggety-wriggety-wrecked-son" => "http://www.reactiongifs.us/wp-content/uploads/2016/02/riggity_wrecked_son_rick_morty.gif",
  "planning-for-failure" => "http://2.media.dorkly.cvcdn.com/78/27/5e7374300a84037f2bdbc061f2e69211.jpg",
  ###
  # Several lines removed for brevity
  ###
  "not-today" => "http://i.imgur.com/kxHrcC5.gif"
}

post '/images' do
  image_response COMMAND_IMAGE_MAPPING[params['text']]
end

But what if our users send an invalid argument to the Slash command? We should give a helpful response to the user, listing all of the valid arguments they could use with the Slash Command. We can do this by simply joining all of the keys of the Hash, and returning that as part of a helpful error message.

COMMAND_IMAGE_MAPPING = {
  "nobody-exists-on-purpose" => "http://cdn.smosh.com/sites/default/files/2015/12/rickmorty15.jpg",
  "wriggety-wriggety-wrecked-son" => "http://www.reactiongifs.us/wp-content/uploads/2016/02/riggity_wrecked_son_rick_morty.gif",
  "planning-for-failure" => "http://2.media.dorkly.cvcdn.com/78/27/5e7374300a84037f2bdbc061f2e69211.jpg",
  ###
  # Several lines removed for brevity
  ###
  "not-today" => "http://i.imgur.com/kxHrcC5.gif"
}

post '/images' do
  if COMMAND_IMAGE_MAPPING.key?(params['text'])
    image_response COMMAND_IMAGE_MAPPING[params['text']]
  else
    images = COMMAND_IMAGE_MAPPING.keys.sort.join(",\n  ")
    string_as_json_response "Cannot find that image #{params['text']}. Full list of images:\n  #{images}"
  end
end

Let's add one more affordance for our users. Let's add a second Slash Command, (/rm-list-images) and a corresdonding API route (/images/all) that just returns the full list of possible images.

post '/images/all' do
  images = COMMAND_IMAGE_MAPPING.keys.sort.join(",\n  ")
  string_as_json_response "Full list of images:\n  #{images}"
end

Just for reference, here are what the two Slash Commands look like in the Slack UI:

Perfect, so our original image commands still work and we've made room for some new commands through our refactor.

Dynamic Memes

Images are all well and good, but we can do better. Let's add a feature to generate dynamic memes, so you can apply Rick and Morty humor to your 9-to-5.

I used the memegenerator.net API and I highly recommend it. It's free and the authentication is dirt simple (albeit with a cleartext password). All you need to do is sign up for an account on the Meme Generator homepage.

In their API documentation, you'll notice that meme templates are referred to as 'generators' while an instance of a particular meme (with custom text) is just referred to as an instance. Now we could use the API to look up the generator ID based on the name of the meme, but people have used all sorts of garbage names like 'Rick Sanchez123.' Instead, I found a few memes I wanted to support in the Meme Generator UI, looked up the generator ID for that specific meme, and recorded it in a Ruby hash.

COMMAND_MEME_MAPPING = {
  "tiny-rick" => {
    "generatorID": 5200542,
    "displayName": "Tiny Rick"
  },
  "get-schwifty" => {
    "generatorID": 5841203,
    "displayName": "Rick Sanchez Schwifty"
  },
  "rick-shrug" => {
    "generatorID": 5733231,
    "displayName": "Rick Sanchez123"
  },
  "rick-scolding-morty" => {
    "generatorID": 6223262,
    "displayName": "Rick Sanchezzz"
  },
  "way-up-your-butt" => {
    "generatorID": 6063115,
    "displayName": "Rick Sanchez 2"
  },
  "get-your-shit-together" => {
    "generatorID": 6168409,
    "displayName": "mortygetyourshittogether"
  },
  "rick-and-morty-couch" => {
    "generatorID": 4875756,
    "displayName": "Rick and Morty Couch"
  },
  "i-just-got-bored" => {
    "generatorID": 4229669,
    "displayName": "Rick and Morty - I'm bored"
  },
  "show-me-what-you-got" => {
    "generatorID": 6873210,
    "displayName": "Show Me What You GotT"
  },
  "bird-person" => {
    "generatorID": 3687578,
    "displayName": "Bird Person Rick and Morty"
  },
  "look-morty" => {
    "generatorID": 5932800,
    "displayName": "Grandpa Rick"
  }
}

In addition to the key, I also retained the original display name for reference later.

Generating a meme is fairly trivial. You just need to make an HTTP GET request to

http://version1.api.memegenerator.net/" +
  "Instance_Create?username=#{MEME_GENERATOR_USERNAME}" +
  "&password=#{MEME_GENERATOR_PASSWORD}&languageCode=en&" +
  "generatorID=#{generator_id}&text0=#{text0}&text1=#{text1}"

where the generatorID is the ID of the meme you want to use, text0 is the top line of text, and text1 is the bottom line of text. To illustrate:

OK, so now we just need another API route and Slash command to handle generating these memes on the fly. For our Slash Command, let's plan on sending those 3 pieces of information as a text argument. Because we'll be sending arbitrary strings of text for our dank memes, let's use '; ' (read: semi-colon, space) to separate those elements (e.g. "meme ID; top text; bottom text").

The other thing we have to consider before writing any code is timeouts. Slash Commands must get a valid HTTP response within three seconds, or they will throw a timeout error. We can't assume the Meme Generator's API or our app will be able to reliably return an image in three seconds. Instead of responding to the original HTTP request, our app will make an asynchronous request to the 'response_url' provided in the Slack payload once the meme generation completes.

With that in mind, let's start with building the API route in Sinatra:

post '/memes' do
  # Grab the entire text from the Slash Command and split the meme,
  # top text, and bottom text from the text argument from Slack
  text_params = params['text'].split('; ')
  # Grab the response URL from the Slack HTTP call. This is the
  # Slack endpoint we'll post the result to
  response_url = params['response_url']

  # If the meme name arg is found in the meme hash...
  if COMMAND_MEME_MAPPING.key?(text_params[0])
    generator_id = COMMAND_MEME_MAPPING[text_params[0]][:generatorID]
    text0 = URI.encode(text_params[1])
    text1 = URI.encode(text_params[2])
    # generate memes
    response = HTTParty.get("http://version1.api.memegenerator.net/" +
      "Instance_Create?username=#{MEME_GENERATOR_USERNAME}" +
      "&password=#{MEME_GENERATOR_PASSWORD}&languageCode=en&" +
      "generatorID=#{generator_id}&text0=#{text0}&text1=#{text1}")
    if response['success']
      post_image_to_response_url response_url, response['result']['instanceImageUrl']
      status 200
    else
      string_as_json_response "Error generating meme"
    end
  # If the meme is not found, return an error message and
  # include full list of available memes
  else
    memes = COMMAND_MEME_MAPPING.keys.sort.join(",\n  ")
    string_as_json_response "Cannot find that meme #{text_params[0]}. Full list of available memes:\n  #{memes}"
  end

end

New users may not know all of the meme options available, so let's also throw in an API route to list all of the available memes.

post '/memes/all' do
  memes = COMMAND_MEME_MAPPING.keys.sort.join(",\n  ")
  string_as_json_response "Full list of memes:\n  #{memes}"
end

Finally, let's create a Slash Command for each of those endpoints in the Slack app management console:

Before we can test these new commands, we need to set the MEME_GENERATOR_USERNAME and MEME_GENERATOR_PASSWORD variables in a safe and secure way. First, let's set them as environment variables. In Heroku, this is as simple as executing:

[user@host ~]$ heroku config:set MEME_GENERATOR_USERNAME=my_username
[user@host ~]$ heroku config:set MEME_GENERATOR_PASSWORD=my_password

Now you can refer directly to those environment variables in-line, like this:

response = HTTParty.get("http://version1.api.memegenerator.net/" +
      "Instance_Create?username=#{ENV['MEME_GENERATOR_USERNAME']}" +
      "&password=#{ENV['MEME_GENERATOR_PASSWORD']}&languageCode=...

Excellent, now we can test our command. Let's look at the GIF demo one more time to see it in action:

OAuth and "Publishing"

In order to publish our Slash Command app on Slack's marketplace or share it with the "Add to Slack" button, our application needs to have an OAuth endpoint that initiates the user authentication to their team, thus authorizing our application to be installed into their Slack team.

In more complex applications (like real time chat bots) we need to store information sent to us in the OAuth request from Slack. In our simple app, we just need to respond to the OAuth request by POSTing to Slack's /api/oauth endpoint.

Let's start be defining our OAuth endpoint in Slack. Let's go back to https://api.slack.com/apps, drill into our app by clicking the app name, and then click the "OAuth & Permissions" on the left. Next, let's define our Redirect URL as 'https://your-domain/oauth' and save.

Next, let's click on the Basic Information tab, and capture the 'client ID' and 'client secret':

Don't be an idiot and commit these to Git like I did. Keep these out of your repo, or put them in an environment variable file and add it to your .gitignore. If someone captured your client ID and secret, they could impersonate your application and wreak havoc.

Once we've got our redirect URL set up and gathered our OAuth client keys, we can construct an easy /oauth route in our Sinatra app. Our route should trigger a POST to the Slack /api/OAuth endpoint. If that post gets an HTTP 200 response, let's just post some simple HTML to show the app was successfully installed. If not, let's dump the response.body to determine the cause of the failure.

get '/oauth' do
  response = HTTParty.post("https://slack.com/api/oauth.access?client_id=#{ENV['SLACK_CLIENT_ID']}&client_secret=#{ENV['SLACK_CLIENT_SECRET']}&code=#{params['code']}")
  responseBody = JSON.parse(response.body)
  if responseBody['ok'] == true
    "<h1>You've just installed the YouPassButter Slack bot! Let's get riggety wrecked!!!</h1>"
  else
    response.body
  end
end

Much like before with our Meme Generator credentials, we have to set these Slack client credentials as environment variables so they can be securely consumed by our app without being in source control (Git).

[user@host ~]$ heroku config:set SLACK_CLIENT_ID=1234567890
[user@host ~]$ heroku config:set SLACK_CLIENT_SECRET=0987654321

Alright, now that we have our OAuth endpoint set up, let's focus on packaging our app.

In order for people to install our app, we need to use either the "Add to Slack" button or publish it on the Slack App Directory. To get published to the App Directory, your app has to be approved by Slack, you have to provide a landing page for the app, and you need some basic customer support mechanisms in place, etc. In short, it's a non-trivial effort.

Instead, we can just create the "Add to Slack" button. The process is much simpler and there's less overhead. Hell, the code is even generated for you automatically just by browsing to https://api.slack.com/docs/slack-button (assuming you're already logged in). Let's do that now. Scroll down to the "Add the Slack Button" section, select your Slash Command app in the dropdown in the top right, and then check only the 'commands' checkbox:

Once you've got the HTML copied, you can drop it into your blog, emails, or a landing page you maintain for the app. Here's the Add to Slack button for the YouPassButter app:

Here's a quick demo of a user clicking the button, authenticating to their Slack team, and getting a successful response from our /oauth endpoint:

Now obviously our confirmation page is a little plain to say the least, but whatever, it works!

Conclusion

That's it for our YouPassButter Slack bot. It fetches images/gifs, generates memes, and can be easily installed by any Slack team. Now for a celebratory demo gif:

23-Jun-2017 Update

I got a friendly email from someone in Australia who wanted to use the YouPassButter bot (which is awesome, love hearing from people who use my code/apps), but the “Add to Slack” button wasn’t working. It would just throw a JSON response in the body complaining about an invalid_client_id (which absolutely wasn’t the problem, because the client ID is immutable for an app and I didn’t change the code). So I started to dig in, reproduced the error, but there was a separate URL in the “Your Apps” page on Slack’s site that worked fine. So after a few minutes of “WTF?” and verifying it worked a few times, I dug into the anchor tag behind the link. Oddly enough, it was the exact same URL from my “Add to Slack” button except it included another URL parameter, install_redirect=general. As a quick experiment, I updated my “Add to Slack” button and it started working! I opened a support case with Slack to find out why this change happened, I’ll let all of you know if/when I hear back.

In the meantime, be wary of this issue and let me know if you find it mentioned in the documentation anywhere.

02-Sep-2017 Update

The issue mentioned in the earlier update was entirely my fault, no issue with Slack or Heroku. The whole install_redirect things was a red herring. In an effort to better format my code and comply with Rubocop, I accidentally introduced additional whitespace. When my Heroku app tried to authenticate to Slack, it was posting to https://slack.com/api/oauth.access?client_id= 1229... instead of https://slack.com/api/oauth.access?client_id=1229.... You can see the issue here: https://github.com/alexdglover/you-pass-butter-bot/commit/cc0318b0e5563b4d454e785978953d00cd451987#diff-cc95738088603531796e0d0f246a5d77L97

Unfortunately right after I solved that issue, I discovered that MemeGenerator.net has changed their API such that they do not return a full URL to the generated image, breaking the meme generation capability. I’m hoping to contact them and find a solution or have them update their API. Until then, I’m disabling the meme generator commands. Please let me know if you have any other issues with the bot!

YouPassButter Slack Bot, Part 1

2017-01-07T17:34:38+00:00

TL;DR Section

Just looking for the app? Just click the "Add to Slack" button to install the YouPassButter bot

Don't care about the context, just looking for a general guide to building a Slack bot with Ruby and Sinatra? Skip down to the "Implementation" section.

Prefer to just go straight to the source? Check out the code on GitHub.

Background/Context

I've wanted to build a Slack bot for a while now, but didn't have any ideas for what the bot should do. Then it hit me - a Rick and Morty bot for getting funny quotes and memes posted to Slack. And what robot better represents Rick and Morty then the butter-passing robot Rick built

I'm also starting a new job in a few days, and so I was looking for an excuse to write something in Ruby.

Finally, like all of my projects, this took a couple of iterations before I was really happy with it, so this post will come in a few parts.

Requirements

As a user, I want to be able to summon a pre-defined Rick and Morty image or gif
As a user, I want to be able to summon a dynamically generated Rick and Morty meme with Slash Command inputs being used to generate meme

Mechanics

Slack bots (and chat bots in general) can be very complex, but they don't have to be. With Slack, the simplest implementation IMO is the "Slash Command" bots. These aren't really "bots" per se, but rather a few pre-defined Slash Commands that call out to a web service endpoint and get a response. When someone installs this "bot" it just makes the Slash Commands available to their team.

Behind the scenes, these Slash Commands are associated with a URL endpoint, e.g. https://my.web.service.com/route. When you execute a Slash Command, Slack makes a GET or a POST to that URL endpoint including a payload that includes, among other things, any arguments that were passed to the slack command.

For example, let's assume our Slack app has "/foo" defined as a Slash Command that is associated with https://my.web.service.com/foo. If a user executes "/foo bar" from the "general" channel, Slack will make a POST request to https://my.web.service.com/foo with the following payload:

token=gIkuvaNzQIHg97ATvDxqgjtO
team_id=T0001
team_domain=example
channel_id=C2147483705
channel_name=general
user_id=U2147483697
user_name=Alex
command=/foo
text=bar
response_url=https://hooks.slack.com/commands/1234/5678

Now, if you just want to make an HTTP request and not get any result back (i.e. kicking off a job), you could simply define the Slash Command and be done with it (assuming the receiving endpoint was already set up).

For all other use cases, your endpoint needs to either

return relevant data in the response body within 3 seconds of the initial request OR
make a delayed response by POSTing to the response_url in the initial request

For our YouPassButter bot, we'll use both of these approaches. We can use the immediate response approach for any existing images or GIFs. For the generated memes, we need to use the delayed response approach because of the latency of generating a new meme each time.

Implementation

This is a fairly easy project - we just need to deploy a web service that can take POST and some arguments, and either respond immediately or make a POST request to a Slack endpoint. As I mentioned, I wanted to brush up on my Ruby so I decided to use Sinatra for this project.

I have built Slack bots previously, but never in Ruby. To get myself grounded, I took a look at this tutorial to get started: https://www.sitepoint.com/building-a-slackbot-with-ruby-and-sinatra/ (Plus I like to give credit to the other blogs/sites that inspire and teach me).

I started by writing a very simple Sinatra app, just to smoke test that everything was working.

Here's the directory structure of the application:

rick-and-morty-bot/
|--app.rb
|--config.ru
|--Gemfile

To start, the Gemfile just contained the 'sinatra' and 'json' gems:

source 'https://rubygems.org'

gem 'sinatra'
gem 'json'

The config.ru is very barebones, just enough to deploy the app to Heroku:

require './app'
run Sinatra::Application

And finally, our initial app.rb:

require 'sinatra'
require 'json'

set :protection, :except => [:json_csrf] # This is a Sinatra setting I needed to address 40x errors

################################################
# Generic routes
################################################

get '/' do
  string_as_json_response "hello world"
end


################################################
# Image related routes
################################################

post '/you-pass-butter' do
  image_responpse "https://media.giphy.com/media/Fsn4WJcqwlbtS/giphy.gif"
end


################################################
# Utility functions
# These functions handle generic formatting of the response body
# to return to Slack
################################################

def image_response url
  message = {
    :response_type => "in_channel",
    :attachments => [
      { :image_url => url }
    ]
  }
  hash_as_json_response message
end

def hash_as_json_response message
  content_type :json
  message.to_json
end

def string_as_json_response message
  content_type :json
  { :text => message }.to_json
end

That's it, super simple. If our Sinatra app receives a POST request at '/you-pass-butter' then our web service will return a JSON response that includes the GIF attachment:

{
  "response_type": "in_channel",
  "attachments": [
    { "image_url": "https://media.giphy.com/media/Fsn4WJcqwlbtS/giphy.gif" }
  ]
}

The "response_type": "in_channel" tells Slack that this message should be shown to everyone in the channel, not just the user who executed the Slack command. On that note, this is probably a good time to link to some of Slack's documentation and resources:

Slash Command Bot Overview

Attaching content and links to messages

Basic Message Formatting

Message Formatting "Playground" (useful for verifying the syntax/structure of your JSON response)

At this point, we can deploy our app to Heroku and smoke test it. By cURL'ing that endpoint, we can verify that the app is working correctly. Adding new routes is super simple, we just follow the same pattern for our "/you-pass-butter" route and provide a different image URL to the image_response method. Let's add a few more routes:

require 'sinatra'
require 'json'

set :protection, :except => [:json_csrf] # This is a Sinatra setting I needed to address 40x errors

################################################
# Generic routes
################################################

get '/' do
  string_as_json_response "hello world"
end


################################################
# Image related routes
################################################

post '/you-pass-butter' do
  image_responpse "https://media.giphy.com/media/Fsn4WJcqwlbtS/giphy.gif"
end

post '/wriggety-wriggety-wrecked-son' do
  image_response "http://www.reactiongifs.us/wp-content/uploads/2016/02/riggity_wrecked_son_rick_morty.gif"
end

post '/planning-for-failure' do
  image_response "http://2.media.dorkly.cvcdn.com/78/27/5e7374300a84037f2bdbc061f2e69211.jpg"
end

post '/where-are-my-testicles-summer' do
  image_response "http://i.imgur.com/xOcpvUM.png"
end

post '/the-answer-is-dont-think-about-it' do
  image_response "http://i.imgur.com/L88mBul.jpg"
end

post '/its-getting-weird' do
  image_response "https://i.imgur.com/bBLnvGu.jpg"
end

post '/i-just-wanna-die' do
  image_response "http://i.imgur.com/CkmvRq8.jpg"
end

post '/wubalubadubdub' do
  image_response "http://i.imgur.com/fub72FV.png"
end


################################################
# Utility functions
# These functions handle generic formatting of the response body
# to return to Slack
################################################

def image_response url
  message = {
    :response_type => "in_channel",
    :attachments => [
      { :image_url => url }
    ]
  }
  hash_as_json_response message
end

def hash_as_json_response message
  content_type :json
  message.to_json
end

def string_as_json_response message
  content_type :json
  { :text => message }.to_json
end

Now that we have our web service up and running, we just need to create the integration on the Slack side. If you're just creating Slash Commands for your team, this is super easy and you don't have to create a true Slack "app." Just log on to your Slack team, click on the dropdown next to your name, choose "Apps & integrations," then search for "Slash Commands" (assuming you don't already have this app installed).

Once you pull up the Slash Commands app, click on "Add Configuration." From there, you can provide whatever keyword you want for your Slash Command, then click "Add Slash Command Integration." On the next screen, provide the command name you want to use, and hit "Add Slash Command Integration."

On the next screen, there is a bunch of useful information about Slash Commands at the top, followed by configuration elements for our Slash Command. Let's change our command name to /you-pass-butter, and we'll set the URL to our application endpoint. In my case, that endpoint is https://rick-and-morty-bot.herokuapp.com/you-pass-butter. Also note that you can use GET or POST, but this only applies to custom Slash Commands you set up as part of your Slack 'team.' If you want to publish your Slash Command as an app, you need to use POST. In our case, we don't care which team the request is originating from, so we can ignore the token. You can also provide help text along with your slack command if you so choose.

That's it! Now you should be able to go to your Slack channel and execute '/you-pass-butter' and get the image returned!

Awesome!

Part 2

In Part 2, we'll cover

handling our second requirement, namely generating dynamic memes and POSTing them back to Slack
packaging our Slash Commands as a Slack app, including the OAuth configuration needed for the "Add to Slack" button
Re-structuring our Slash Commands and routes to get more functionality

Stay tuned!

Issues with AWS CodeDeploy and CIS hardening

2015-10-22T03:23:25+00:00

I've been using AWS CodeDeploy for some time now and it's really an awesome service. If you're hosting your application in AWS and not using CodeDeploy you're missing out.

If you want to use CodeDeploy yet but you just don't know how, check out my in-depth AWS CodeDeploy course.

I had been successfully using CodeDeploy for an app at work for a few months when I was told we had to switch to using Linux AMI's that were 'hardened' to meet CIS benchmarks for security. No problem right? I redeployed the app stack with the CIS-hardened AMI and worked through a few bugs that came out of it.

However, when I went tried to execute my CodeDeploy deployment it died with an error claiming that it couldn't access the "patch.sh" file due to permissions. I logged on to the instance, cd'd into the CodeDeploy directory, and didn't notice anything obviously wrong with the permissions.

I took a look at the appspec.yml to see what might be different about this one script that was failing. Here's a mock-up of the appspec.yml similar to the one I was using:

version: 0.0
os: linux

hooks:
  BeforeInstall:
    - source: scripts/deregister-from-elb.sh
      timeout: 1200
    - source: scripts/stop-app.sh
      timeout: 1200
  AfterInstall
    - source: scripts/patch.sh
      timeout: 600
      runas: ec2-user
    - source: scripts/start-app.sh
      timeout: 600
      runas: ec2-user
    - source: scripts/register-with-elb.sh
      timeout: 1200

That's it - patch.sh is the first script executed as ec2-user. But this revision worked before, what changed?

One of the many changes required to meet the CIS benchmarks includes changing the umask so "that files created by daemons will not be readable, writable or executable by any other than the group and owner of the daemon process..." Because the CodeDeploy daemon runs as root, any files created (including every file and script that is part of your CodeDeploy revision) are owned by root:root with permissions 750. So any script being called by a user that's not root (or in the root group) can't execute the scripts.

To workaround this, you can simply add another script to update the permissions of the CodeDeploy deployment-root before you execute any scripts as a non-root user.

Here's the updated appspec.yml:

version: 0.0
os: linux

hooks:
  BeforeInstall:
    - source: scripts/update-deployment-root-perms.sh
      timeout: 60
    - source: scripts/deregister-from-elb.sh
      timeout: 1200
    - source: scripts/stop-app.sh
      timeout: 1200
  AfterInstall
    - source: scripts/patch.sh
      timeout: 600
      runas: ec2-user
    - source: scripts/start-app.sh
      timeout: 600
      runas: ec2-user
    - source: scripts/register-with-elb.sh
      timeout: 1200

And the update-deployment-root-perms.sh script:

#!/bin/bash

echo "Here's some meaningful message before we attempt to update the permissions of the deployment-root directory and all child files and folders"

chmod -R 755 /opt/codedeploy-agent/deployment-root

echo "Here's some meaningful message to confirm the script successfully executed the chmod command"

That's it - a simple solution for an obscure problem.

As always, I hope this post has been helpful and let me know if you have any questions.

Arduino Project 7: Buzzword

2015-08-23T18:13:04+00:00

Here's a quick and easy project you can do with minimal hardware components. I call this project/device "Buzzword" but it's basically the same game and concept as Catch Phrase. In Catch Phrase, you have a device that shows you random words. You can say anything to help your team guess the word as long as you don't say any part of the "catch phrase" itself, say "it rhymes with ____ ," or otherwise cheat. Once your team correctly guesses the word, you pass the device to the opposing team. This goes on until you run out of time. Whichever team is holding the device when time is up loses.

My game is very similar, except teams compete by seeing how many points they can get rather than simply whoever is holding the device at the end of the round. In addition, since you are providing your own list of words for the game in the code, you can update it once you get bored with the words or add new pop culture references.

Here's a quick demo:

Or for those who don't want to wait for a video to load, here are a few screenshots that should give you an idea of how it works.

Here's the initial startup screen
Once you click the button to start, a 60 second timer starts and you're shown the first word. The list of words is shuffled every time the game starts. At this point, if you're holding the device, you can pass if it's too hard, or start throwing out clues. If someone guesses correctly, you hit the +1 button which will increment your team's score and load up the next buzzword for the next player
This continues...
Until finally you run out of time and you get your final score

Components Needed

Arduino Uno
Arduino Uno compatible 2.8" TFT Touch Shield V2.0 from Seeedstudio (you could use other boards but may need different libraries)
9v Battery Power Pack with DC male barrel connection

Construction/Assembly

This one's going to be really simple. The trickiest part is going to be loading the libraries before you upload your sketch.

Connect the TFT Touch Shield to the Arduino. It only goes on one way, so you can't really mess this up. If it doesn't slip into place immediately, check that the pins are lining up and bend them as necessary. If all of the pins line up well, don't be afraid to use a little force.
Download the relevant libraries and my sketch
1. Seeedstudio Touchscreen Library
2. The Buzzword Sketch (includes additional libraries)
Connect the Arduino Uno to your computer with the USB cable.
Open the Arduino IDE. If you don't have it already, download it and install before proceeding.
Add the Seeed-Studio Touch_Screen_Driver and the TFTv2_Touch_Shield-Text_Direction libraries
1. In the Arduino IDE menu, go to Sketch --> Import Library... --> Add Library...
2. Find the Touch_Screen_Driver-master library you downloaded earlier in the file system browser and select it. The directory should contain SeeedTouchScreen.cpp file, a SeeedTouchScreen.h file, and a keywords.txt file at a minimum.
3. Repeat the above steps for the TFTv2_Touch_Sheild-Text_Direction library
Open the Buzzword sketch and the font.c file
1. In the Arduino IDE menu, go to File --> Open...
2. Find the Buzzword.ino file in the Buzzword directory and select it. The main editor window in the Arduino IDE should now be populated with the code
3. In the Arduino IDE menu, go to Sketch --> Add File...
4. Find the font.c in the root of the Buzzword directory and select it. A second tab should pop up and be populated with some text.
Compile the code to verify
1. Click on the buzzword tab in the Arduino IDE
2. Click the icon in the top left with a checkmark symbol - this will compile the code and verify there are no compilation issues.
If all is well at this point, upload to your Arduino by clicking the icon in the top left with a right arrow icon. This will re-compile the code and attempt to write it to the Arduino itself. The screen should light up and display the start button, and should be ready to rock at this point.
If the device is working as expected, go ahead and disconnect the USB, plug in the battery pack, and secure the battery pack (two-sided tape or zip ties are good temporary solutions).
Play Buzzword!

Arduino Project 6: Unlock it with Fire

2015-07-12T20:10:09+00:00

A couple years ago I built a novelty lock box that used an Arduino and an accelerometer so that you had to tilt the box in the correct sequence to unlock it. The prototype was pretty rough, so I eventually refined the design and built a Mark II version of it.

This project is my third iteration of the novelty lockbox. The majority of the design has remain unchanged, except for this project I used a "flame sensor" instead of an accelerometer. Instead of tilting the box in a particular sequence, the only way to open the box is to use an open flame as the "key." Here's a quick video to demonstrate the finished product:

Component List

A 3-5.5v "flame sensor"
An Arduino Uno (or materials to build your own breakout board)
A DC power supply with both 5v and 12v output (I used an old molex power supply)
A single relay
A 12v door latch

Testing

Before you do anything with the box, we want to test and calibrate the flame sensor. I followed this Instructable to get acquainted with the flame sensor and what kind of values it returns. Note that the author is mapping the full range of values (0-1024) to just 4 values to match his sketch. Start by modifying this block:

void loop() {
  // read the sensor on analog A0:
  int sensorReading = analogRead(A0);
  // map the sensor range (four options):
  // ex: 'long int map(long int, long int, long int, long int, long int)'
  int range = map(sensorReading, sensorMin, sensorMax, 0, 3);

by adding a line for logging the raw sensor reading, like this:

void loop() {
  // read the sensor on analog A0:
  int sensorReading = analogRead(A0);
  Serial.println(sensorReading);

  // map the sensor range (four options):
  // ex: 'long int map(long int, long int, long int, long int, long int)'
  int range = map(sensorReading, sensorMin, sensorMax, 0, 3);

Now you can wire up the sensor, upload your sketch, and experiment with what kind of values you get with an open flame at different distances.

One quick clarification - I refer to this device as a "flame sensor" because that's how its marketed, but in reality it's just a photoresistor that has a filtering film around it that only allows infrared light to pass through. The voltage in the photoresistor circuit will change with the amount of light, and the Arduino analog pin will convert that voltage value to an integer between 0 and 1024. This means any infrared light will affect the sensor, including sunlight, TV remotes that use infrared, even something that is very hot could theoretically give off enough infrared to affect the sensor. If you're getting strange values while testing your sensor, be sure that you don't have another source of infrared throwing you off.

Once you're confident your sensor works and you have some idea of what values it returns, you're ready to build out the lock box components.

Assembly

Note: To save on cost, I've been building Arduino compatible breadboards using standalone ATmega328 chips. The instructions will still work for an Arduino Uno, but the pictures won't be as helpful. Check out Arduino Project 4: Tilt to Unlock to see pictures for a similar assembly using an Arduino Uno.

Connect the 5v and ground pins (may be marked as '+' and 'G') on your flame sensor to the corresponding pins on the Arduino or bread board. Connect the A0 pin (the analog pin) on the flame sensor to the Analog Input 0 pin on the Arduino (green wire in the photo below, pin 23 on an ATmega328). Use male-to-female jumper wires to make this easier.
Connect the 5v and ground pins (may be marked as 'VCC' and 'GND') on the relay to the corresponding pins on the Arduino or bread board. Connect the 'IN' pin to Digital Pin 8 (yellow wire in the photo above, pin 14 on an ATmega328). The digital pin will close the relay when voltage is applied, closing the circuit for our door latch and opening the lock. Use male-to-female jumper wires to make this easier.
The door latch has two wires, they are arbitrary (either one can be powered or ground). Connect one of them to the ground wire from the power supply. Connect the other wire to the normally open ('NO') socket on the relay. Connect the 12v wire from the power supply to the always-closed or common socket on the relay (it's almost always the middle one). Here's a closeup of the wiring from Arduino Project 4:

So our project should be all wired up and ready for a smoke test. Go ahead and connect the power and make sure nothing starts smoking or getting too hot.
If you're still reading that means nothing melted or started smoking, which is great. At this point we're ready to upload a sketch and test. Go ahead and download the sketch and upload it to your Arduino. If all goes well, the door latch should open when you provide the infrared light. Here's a quick test video:

Now we're ready to assemble the box.
Drill a hole in the back of the cigar box just big enough to run your power supply wires through, including the insulation. Run your wire through and let it hang outside of the box for later (this will also prevent you from locking yourself out of the box by accident).
Remember how I mentioned other sources of infrared can interfere with your sensor? We can limit this effect by "tunneling" or narrowing the sensor's field of view. Start off by drilling a small hole in your cigar box. A 1/4" inch drillbit should work well for this. Make sure your hole placement lines up with a reasonable place to mount your flame sensor inside of the box. To be discreet, I drilled my hole on the bottom left of the "face" of the cigar box and mounted the sensor using double sided tape.
If you're starting with a "clam-shell" style box (cigar boxes work great), you can follow my solution for the door latch. Secure the 12v door latch to the top of the cigar box with 1/4" screws. Next, take a sawtooth picture hangar and nail it into place so it lines up with the 12v door latch. Obviously hammering inside of a cigar box can be difficult - you can also use a pair of pliers to squeeze the nail into place.
At this point, things are going to start getting tight so go ahead and re-wire everything outside of the box. Once you're finished, mount your Arduino (or breadboard) and relay inside of the box. Try to keep all of these components mounted off to one side to save as much usable space as possible.

I mounted all of my components to the side of the cigar box. Make sure your 12v door latch won't hit your other components when you close the box.

That's it, your lock box is complete! As always I hope you enjoyed the post and I look forward to seeing what you build!

GitHub Repo

Share Your Salary App Part 3

2015-02-23T16:30:16+00:00

So in Part 1 we built the core classes for our app and we exposed those classes as a RESTful web service in Part 2. Now, we'll build a UI and wrap up the project.

Feel free to check out the application first or pull up the source in another window.

Demo GitHub Repo

The Fat Free Framework configuration allows you to specify a document root for the front-end pages. By default, this is set to the 'ui' directory in the root of the F3 directory structure. This will be our working directory for building the UI. The F3 package will have a few pages, CSS, and images in the 'ui' directory, you can delete them all if you want (I kept the welcome.htm file just out of convenience).

Now I'm not a good UI developer, so I tend to use the Bootstrap framework (v2.3.2) for the front-end of my applications. Go ahead and download the Bootstrap package and unzip it in the 'ui' directory. The 'ui' directory structure should look like this:

ui/
- js/
  - bootstrap.js
  - bootstrap.min.js
  - jquery.js
- css/
  - base.css
  - bootstrap-responsive.css
  - bootstrap-responsive.min.css
  - bootstrap.css
  - bootstrap.min.css
  - theme.css
- img/
  - glyphicons-halflings-white.png
  - glyphicons-halflings.png
- welcome.htm (from the f3 package)

Now we have the components that we need to start building pages. Let's start by building a layout.htm file - this will be the base HTML file (with a common header, footer, styles and script tags) that will be populated with content for each page. These types of pages/files are referred to as templates.

Again... I'm a terrible UI developer so I blatantly 'borrowed' the majority of this page from one of the Bootstrap examples.

<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>ShareYourSalary.Info</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="">
    <meta name="author" content="">
    <script src="/ui/js/jquery.js"></script>
    <script src="/ui/js/bootstrap.js"></script>
    <script type="text/javascript">
      $(document).ready(function(){
        $('[data-toggle="popover"]').popover({
    placement : 'top'
        });
      });
    </script>

    <!-- Le styles -->
    <link href="/ui/css/bootstrap.css" rel="stylesheet">
    <style type="text/css">
      body {
        padding-top: 20px;
        padding-bottom: 60px;
      }
      /* Custom container */
      .container {
        margin: 0 auto;
        max-width: 1000px;
      }
      .container > hr {
        margin: 60px 0;
      }
      /* Main marketing message and sign up button */
      .jumbotron {
        margin: 80px 0;
        text-align: center;
      }
      .jumbotron h1 {
        font-size: 100px;
        line-height: 1;
      }
      .jumbotron .lead {
        font-size: 24px;
        line-height: 1.25;
      }
      .jumbotron .btn {
        font-size: 21px;
        padding: 14px 24px;
      }
      /* Supporting marketing content */
      .marketing {
        margin: 60px 0;
      }
      .marketing p + h4 {
        margin-top: 28px;
      }
      /* Customize the navbar links to be fill the entire space of the .navbar */
      .navbar .navbar-inner {
        padding: 0;
      }
      .navbar .nav {
        margin: 0;
        display: table;
        width: 100%;
      }
      .navbar .nav li {
        display: table-cell;
        width: 1%;
        float: none;
      }
      .navbar .nav li a {
        font-weight: bold;
        text-align: center;
        border-left: 1px solid rgba(255,255,255,.75);
        border-right: 1px solid rgba(0,0,0,.1);
      }
      .navbar .nav li:first-child a {
        border-left: 0;
        border-radius: 3px 0 0 3px;
      }
      .navbar .nav li:last-child a {
        border-right: 0;
        border-radius: 0 3px 3px 0;
      }
    </style>
    <link href="/ui/css/bootstrap-responsive.css" rel="stylesheet">

    <!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
    <!--[if lt IE 9]>
      <script src="ui/js/html5shiv.js"></script>
    <![endif]-->

    <!-- Fav and touch icons -->
    <link rel="apple-touch-icon-precomposed" sizes="144x144" href="/ui/ico/apple-touch-icon-144-precomposed.png">
    <link rel="apple-touch-icon-precomposed" sizes="114x114" href="/ui/ico/apple-touch-icon-114-precomposed.png">
    <link rel="apple-touch-icon-precomposed" sizes="72x72" href="/ui/ico/apple-touch-icon-72-precomposed.png">
    <link rel="apple-touch-icon-precomposed" href="/ui/ico/apple-touch-icon-57-precomposed.png">
    <link rel="shortcut icon" href="/ui/ico/favicon.png">
  </head>
  <body>
    <div class="container">
      <div class="masthead">
        <h1 class="muted">ShareYourSalary</h1>
        <div class="navbar">
          <div class="navbar-inner">
            <div class="container">
              <ul class="nav">
                <li class="active"><a href="/">Home</a></li>
                <li><a href="/swagger/dist/index.html">API Documentation</a></li>
                <li><a href="https://github.com/alexdglover/shareyoursalary">Github Project</a></li>
              </ul>
            </div>
          </div>
        </div><!-- /.navbar -->
      </div>

  <!-- Put content here -->
  <?php echo $this->render(Base::instance()->get('content')); ?>
      <hr>
      <div class="footer">
        <p>&copy; alexdglover 2014</p>
      </div>
    </div> <!-- /container -->
    <!-- Le javascript
    ================================================== -->
    <!-- Placed at the end of the document so the pages load faster -->
    <script src="/ui/js/jquery.js"></script>
    <script src="/ui/js/bootstrap.js"></script>
  </body>
</html>

Now the important piece we've added here is between the 'masthead' and 'footer' divs:

<!-- Put content here -->
<?php echo $this->render(Base::instance()->get('content')); ?>

Now we can just set the 'content' variable in our F3 code to populate the body of the template and we will get consistent and clean looking pages. That's the whole purpose of templates and template engines - build a solid base page (or pages) and just populate the contents dynamically. This makes for great code re-use and you don't have to propagate changes across dozens of files.

Static Pages

For populating the content with a static page, we don't really need a Controller class, we just need F3's routing functionality to set the 'content' variable to a static HTML file. The ShareYourSalary app home page is going to be static, so let's use that as an example.

For the home page, we'll want a simple explanation of what the site/app does and an entry form for users to create a new survey. Our entry form will POST values to the API using jQuery and route us to a result page if it's successful. Remember, we don't have to worry about JavaScript or CSS references because our layout.htm already includes all of that. Let's take a look at the home page (welcome.htm):

<div class="row">
  <div class="span5">
    <h2>Share your salary!</h2>
    <p>Have you ever worried that your colleagues, doing the same job, make more money than you? According to a recent Glassdoor survey, 39% of Americans feel they're underpaid.</p>
    <p>Or maybe you're curious how you rank compared to your high school class?</p>
    <p>Maybe someone on your team is getting underpaid, but you're not comfortable talking about salaries.</p>
    <p>Then create a free, anonymous salary survey!</p>
  </div>
  <div class="span5">
    <form>
      <fieldset>
        <legend>Create a survey</legend>
        <label>Survey Name</label>
        <input class="input-xlarge" name="surveyName" type="text" placeholder="Employer, team name, or totally arbitratry" />
        <label>Currency</label>
        <select class="input-medium" name="currency">
          <option value="" disabled="">Select Currency</option>
          <option value="AUD">Australian Dollar</option>
          <option value="BRL">Brazilian Real </option>
          <option value="CAD">Canadian Dollar</option>
          <option value="CZK">Czech Koruna</option>
          <option value="DKK">Danish Krone</option>
          <option value="EUR">Euro</option>
          <option value="HKD">Hong Kong Dollar</option>
          <option value="HUF">Hungarian Forint </option>
          <option value="ILS">Israeli New Sheqel</option>
          <option value="JPY">Japanese Yen</option>
          <option value="MYR">Malaysian Ringgit</option>
          <option value="MXN">Mexican Peso</option>
          <option value="NOK">Norwegian Krone</option>
          <option value="NZD">New Zealand Dollar</option>
          <option value="PHP">Philippine Peso</option>
          <option value="PLN">Polish Zloty</option>
          <option value="GBP">Pound Sterling</option>
          <option value="SGD">Singapore Dollar</option>
          <option value="SEK">Swedish Krona</option>
          <option value="CHF">Swiss Franc</option>
          <option value="TWD">Taiwan New Dollar</option>
          <option value="THB">Thai Baht</option>
          <option value="TRY">Turkish Lira</option>
          <option value="USD" selected="YES">U.S. Dollar</option>
        </select>
        <label>Period</label>
          <select class="input-medium" name="period">
            <option value="Per Year">Per Year</option>
            <option value="Per Hour">Per Hour</option>
          </select>
        <label>Minimum number of responses <i class="icon-question-sign" data-toggle="popover" data-placement="top" title="Miniumum number of responses" data-content="In order to ensure anonymity, salary survey results won't be published until a minimum number of salary responses are submitted. Set this to '0' if you don't care about anonymity."></i></label>
        <input class="input-mini" name="minEntries" type="number" value=5 />
        <p><input id="submit" type="submit" class="btn btn-success" value="Submit" /></p>
      </fieldset>
    </form>
  </div>
</div>
<script>
  $( "form" ).on( "submit", function( event ) {
    event.preventDefault();
    var datastring = $( "form" ).serialize();
    $.ajax({
      type: "POST",
      url: "/api/v1/survey/AddSurvey",
      data: datastring,
      dataType: "json",
      success: function(data) {
        window.location.href = "/thanks/" + data.URLName;
      },
      error: function(){
        alert('Oops... something went wrong. Please try again.');
      }
    });    
  });
</script>

Perfect. The default index.php that's included in the F3 package will include a route for "GET /" that we can use. This section should already exist in the index.php if you're using the default file that came with the F3 zip file, but just confirm that these lines exist in your index.php:

$f3->route('GET /',
  function($f3) {
    ...
    // lots of default F3 code
    ...
    $f3->set('content','welcome.htm');
    echo View::instance()->render('layout.htm');
  }
);

If you want to use a different file name for your home page (instead of welcome.htm) just change it in this line of the index.php file. Now if we browse to our site, we should see our home page:

Now we can partially test this form if we want. If we submit a survey and the API processes it successfully, we'll get routed to a page that doesn't exist yet (that's OK, that's expected behavior). But how do we know the submission worked? We can go back to our API test page we created in Part 2 and execute a getByName API call.

If we tried to create a survey called 'demo' then all we need to do is execute the /survey/getByName/{name} API with that value. You can do this in Swagger, or since it's just a GET request, you can do it directly in your browser. Just browse to /api/v1/survey/getByName/demo and you should see the JSON response.

OK, at this point we should have a static home page and a working form. Next, we'll have to create dynamic pages for each survey that's created.

Dynamic Pages

For populating dynamic content, you can write your logic directly in F3's index.php, abstract it into individual classes, or you can do what I did and consolidate it all into one PageGenerator class. This will simplify our F3 routes (more on this later) and manage all of our dynamic views in one place.

For this project, we need 3 pages or Views (excluding the static home page):

A page users hit after creating a survey. We'll call this 'thanks' as we'll be thanking users for creating a survey and enable them to invite others to participate
A page for users who've been invited to participate in an existing survey. We'll call this 'addResponse' because ultimately that's the only purpose for this page
A page for users (both invitees and the original survey creator) to review the results of the survey. We'll call this 'report'

First, let's construct our controller that will get all of the relevant variables for these pages and then set the content. Let's review the code:

<?php
class PageGenerator {
  // Instantiate variables
  var $surveyName;
  var $currency;
  var $period;
  var $minentries;
  private $db;

  // Construct the PageGenerator object with a Database object
  public function __construct(){
    $this->db = new Database();
  }

  // Utility function to convert survey names to a URL-friendly name
  private function convertNameToURLName($string) {
    //Lower case everything
    $string = strtolower($string);
    //Make alphanumeric (removes all other characters)
    $string = preg_replace("/[^a-z0-9_\s-]/", "", $string);
    //Clean up multiple dashes or whitespaces
    $string = preg_replace("/[\s-]+/", " ", $string);
    //Convert whitespaces and underscore to dash
    $string = preg_replace("/[\s_]/", "-", $string);
    return $string;
  }


  // Function for populating the 'thanks' page (after a user creates a survey). This collects all of the relevant survey information into the 'survey' JSON object and sets the 'content' variable to the code in the thanks.htm file

  public function thanks($f3,$args){
    $surveys = $this->db->get_collection('surveys');
    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($args['name']);
    $query = array('URLName' => $URLName);
    $cursor = $surveys->find($query);
    foreach ($cursor as $doc){
      $f3->set('survey',$doc);                  
    }
    $f3->set('content','thanks.htm');
    echo View::instance()->render('layout.htm');
  }


  // Function for populating the 'addResponse' page (the page invited users see). This collects all of the relevant survey information into the 'survey' JSON object and sets the 'content' variable to the code in the addResponse.htm file

  public function addResponse($f3,$args){
    $surveys = $this->db->get_collection('surveys');
    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($args['name']);
    $query = array('URLName' => $URLName);
    $cursor = $surveys->find($query);
    foreach ($cursor as $doc){
      $f3->set('survey',$doc);                  
    }
    $f3->set('content','addResponse.htm');
    echo View::instance()->render('layout.htm');
  }


  // Function for populating the 'report' page. This collects all of the relevant survey information into the 'survey' JSON object and sets the 'content' variable to the code in the report.htm file

  public function report($f3,$args){
    $surveys = $this->db->get_collection('surveys');
    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($args['name']);
    $query = array('URLName' => $URLName);    
    $cursor = $surveys->find($query);
    foreach ($cursor as $doc)
    {
      if (count($doc['responses']) < $doc['minEntries'])
      {
        $f3->set('minEntriesMet', false);
      }
      else
      {
        asort($doc['responses']);
        $f3->set('minEntriesMet', true);
        $averageSalary = round((array_sum($doc['responses']))/(count($doc['responses'])),2);
        $f3->set('averageSalary',$averageSalary);
        $percentDeltas = [];
        foreach($doc['responses'] as $response)
        {
          array_push($percentDeltas, (round((($response-$averageSalary)/$averageSalary),2) ) );
        }
        asort($percentDeltas);
        $f3->set('percentDeltas',$percentDeltas);
        // Median code borrowed from http://www.mdj.us/web-development/php-programming/calculating-the-median-average-values-of-an-array-with-php/      
        sort($doc['responses']);
        $count = count($doc['responses']); //total numbers in array
        $middleval = floor(($count-1)/2); // find the middle value, or the lowest middle value
        if($count % 2) { // odd number, middle is the median
          $median = $doc['responses'][$middleval];
        } else { // even number, calculate avg of 2 medians
          $low = $doc['responses'][$middleval];
          $high = $doc['responses'][$middleval+1];
          $median = (($low+$high)/2);
        }
        $f3->set('median',$median);
      } //End of minEntriesMet==true block
      $f3->set('survey',$doc);      
    }
    $f3->set('content','report.htm');
    echo View::instance()->render('layout.htm');
  }
}
?>

As you can see, the functions follow a very similar pattern - we get a survey name input (via F3's $args variable), grab the relevant survey data from the DB, set one (or more) JSON objects to be consumed in the page, and set the 'content' variable to the appropriate page. From inside of the View, we'll grab the individual elements out of the array and use it in the display or JavaScript.

Now for the Views. Let's walk through the app as a user would, just for intuitiveness. Our user has created a survey successfully, so let's thank them for creating it and show them the details of the survey so they can confirm everything is right. Let's also add a mechanism for the user to invite their friends and colleagues through social media. Our survey creator might also be participating, so let's enable them to add a response right away as well. Finally, we need to provide a link to the survey report where users can view the results. Remember, our PageGenerator controller is going to pass us all of the data we need in one array object titled $survey. Let's take a look at the HTML:

<h2>Thanks for creating a survey!</h1>
  <div class="row">
  <div class="span5">
    <legend>Survey Details</legend>
    <p><strong>Survey name</strong><br /><?php echo $survey['name']; ?></p>
    <p><strong>Salary currency</strong><br /><?php echo $survey['currency']; ?></p>
    <p><strong>Salary period</strong><br /><?php echo $survey['period']; ?></p>
    <p><strong>Minimum number of responses before results published</strong><br /><?php echo $survey['minEntries']; ?></p>
    <p><strong>Current number of responses</strong><br /><?php echo count($survey['responses']); ?></p>
  </div>
  <div class="span5">
    <legend>Next Steps</legend>
    <ol>
      <li>Invite others to participate by sharing<br />
      <a target="_blank" href="<?php echo "http://www.facebook.com/sharer/sharer.php?u=" . "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName']; ?>"><img src="/ui/img/facebook_001.jpg" style="width:32px; height:32px;" /></a>
      <a target="_blank" href="<?php echo "https://twitter.com/intent/tweet?text=How%20much%20do%20you%20make%3F&url=" . "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName']; ?>"><img src="/ui/img/Twitter_001.jpg" style="width:32px; height:32px;" /></a>
      <a target="_blank" href="<?php echo "http://www.linkedin.com/shareArticle?mini=true&url=" . "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName'] . "&title=Salary%20Survey"; ?>"><img src="/ui/img/Linkedin_001.jpg" style="width:32px; height:32px;" /></a>
      <a href="<?php echo "http://pinterest.com/pin/create/button/?url=" . "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName']; ?>"  class="pin-it-button" count-layout="horizontal"><img src="/ui/img/Pinterest_001.jpg" style="width:32px; height:32px;" /></a>
      <br />
      Or by sending them this link:<br /><a href="<?php echo "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName']; ?>"><?php echo "http://$_SERVER[HTTP_HOST]" . "/addResponse/" . $survey['URLName']; ?></a></li>
      <li>Enter your salary<br />
        <form>
        <div class="input-append">
          <input name="response" type="text" pattern="[1-9]{1}[0-9]*\.?[0-9]{0,2}" placeholder="Enter your salary" />
          <span class="add-on"><?php echo $survey['currency'] ?>&nbsp;<?php echo $survey['period'] ?></span>
        </div>
        <input id="submit" type="submit" class="btn btn-success" value="Submit" />
        </form>
      </li>
      <li>
        Once the minimum number of entries are met, see the salary analysis report here: <a href="<?php echo "http://$_SERVER[HTTP_HOST]" . "/report/" . $survey['URLName']; ?>"><?php echo "http://$_SERVER[HTTP_HOST]" . "/report/" . $survey['URLName']; ?></a>
      </li>
    </ol>
  </div>
</div>
<script>
  $( "form" ).on( "submit", function( event ) {
    event.preventDefault();
    var datastring = $( "form" ).serialize();
    $.ajax({
        type: "POST",
        url: "/api/v1/survey/addResponse/<?php echo $survey['URLName'] ?>",
        data: datastring,
        dataType: "json",
        success: function(data) {
      alert("Thanks! You've successfully entered your salary information.");
        },
        error: function(){
              alert('Oops... something wrong');
        }
    });    
  });
</script>

One point of interest - you may have noticed the "pattern" property set on the one ofthe HTML inputs. This is a regular expression that tells the input field what valuesare allowed. Let's examine this pattern by breaking it up:
pattern="[1-9]{1}[0-9]*\.?[0-9]{0,2}"

[1-9]{1} - this means we want one and only one character, and it has to be a digit 1-9. This prevents users from submitting a value that starts with a leading zero.

[0-9]* - this means any character 0-9, an unlimited number of times.

\.? - this means we're expecting zero or one period. This allows our users to enter a decimal point if they want, but at most one.

[0-9]{0,2} - this means we'll accept zero, one, or two of any characters between 0 and 9.

What this all boils down to is a user can enter any number with no decimal places, a single decimal place, or two decimal places, and nothing else. No symbols, letters, or illegitimate formats. This ensures we have clean valid numbers to do mathematical operations on later for the 'report' page.

The 'addResponse' page is simpler and re-uses a lot of the same code as the 'thanks' page, so let's jut jump right into it.

<h2>You've been invited!</h1>
<div class="row">
  <div class="span5">
    <legend>Wait, what's going on?</legend>
    <p>You've been invited to participate in an anonymous survey about salaries. Someone has already created the survey and invited you and others to participate. Your salary information is totally anonymous - no login or email address is required. To ensure anonymity, the results of the survey aren't published until a minimum number of responses are made (see below).</p>

    <p><strong>Survey name</strong><br /><?php echo $survey['name']; ?></p>
    <p><strong>Minimum number of responses before results published</strong><br /><?php echo $survey['minEntries']; ?></p>
    <p><strong>Current number of responses</strong><br /><?php echo count($survey['responses']); ?>
  </div>
  <div class="span5">
    <legend>Next Steps</legend>
    <ol>
      <li>Enter your salary<br />
        <form>
        <div class="input-append">
          <input name="response" type="text" pattern="[1-9]{1}[0-9]*\.?[0-9]{0,2}" placeholder="Enter your salary" />
          <span class="add-on"><?php echo $survey['currency'] ?>&nbsp;<?php echo $survey['period'] ?></span>
        </div>
        <p><input id="submit" type="submit" class="btn btn-success" value="Submit" /></p>
        </form>
      </li>
      <li>
        Once the minimum number of entries are met, see the salary analysis report here: <a href="<?php echo "http://$_SERVER[HTTP_HOST]" . "/report/" . $survey['URLName']; ?>"><?php echo "http://$_SERVER[HTTP_HOST]" . "/report/" . $survey['URLName']; ?></a>
      </li>
    </ol>
  </div>
</div>
<script>
  $( "form" ).on( "submit", function( event ) {
    event.preventDefault();
    var datastring = $( "form" ).serialize();
    $.ajax({
        type: "POST",
        url: "/api/v1/survey/addResponse/<?php echo $survey['URLName'] ?>",
        data: datastring,
        dataType: "json",
        success: function(data) {
      alert("Thanks! You've successfully entered your salary information.");
        },
        error: function(){
              alert('Oops... something wrong');
        }
    });    
  });
</script>

Easy right? OK, the last page is the 'report' page where we'll give some basic statistics from the survey results and a nice little chart. The statistics are pretty easy to calculate, and for the chart we'll use Google Charts API via JavaScript. Let's take a look.

<h2>Salary Analysis for Survey '<?php echo $survey['name']; ?>'</h1>
<div class="row">
  <?php if($minEntriesMet) { ?>
  <div class="span5">
    <legend>Survey Details</legend>
    <p><strong>Survey name</strong><br /><?php echo $survey['name']; ?></p>
    <p><strong>Average salary</strong><br /><?php echo $averageSalary ?> <?php echo $survey['currency']; ?> <?php echo $survey['period']; ?></p>
    <p><strong>Median salary</strong><br /><?php echo $median ?> <?php echo $survey['currency']; ?> <?php echo $survey['period']; ?></p>
    <!--<p><strong>Salary currency</strong><br /><?php echo $survey['currency']; ?></p>
    <p><strong>Salary period</strong><br /><?php echo $survey['period']; ?></p>-->
    <p><strong>Minimum number of responses before results published</strong><br /><?php echo $survey['minEntries']; ?></p>
    <p><strong>Current number of responses</strong><br /><?php echo count($survey['responses']); ?>
  </div>
  <div class="span5">
    <legend>Relative Salary Comparison</legend>
    <div id="chart_div" style="width: 100%; height: 300px;"></div>
  </div>
  <?php } else { ?>
  <div class="span12">
    Uh oh, minimum number of responses haven't been met yet. Minimum number of responses is <?php echo $survey['minEntries']; ?> and there have been <?php echo count($survey['responses']); ?> responses so far.<br /><br />No report data available.
  </div>
  <?php } ?>
</div>
<script type="text/javascript" src="https://www.google.com/jsapi?autoload={'modules':[{'name':'visualization','version':'1','packages':['corechart']}]}"></script>
<!--<div id="chart_div" style="width: 100%; height: 500px;"></div>-->
<script>
  google.setOnLoadCallback(drawChart);
  function drawChart() {
    var dataTable =  new google.visualization.DataTable();
    dataTable.addColumn('number', 'Salary');
    dataTable.addColumn('number', '% Delta');
    <?php for($i=0; $i < count($survey['responses']); $i++){
      echo 'dataTable.addRow(['. $survey['responses'][$i]. ',' . $percentDeltas[$i] . ']);';
    };?>

    var formatter = new google.visualization.NumberFormat({
        fractionDigits: 2,
        prefix: '$'
    });

    formatter.format(dataTable, 0); // Apply formatter to salary column.

    var formatter = new google.visualization.NumberFormat({
        fractionDigits: 2,
        pattern: '###%'
    });

    formatter.format(dataTable, 1); // Apply formatter to percent delta column.

    var options = {
      title: 'Salary comparison',
      hAxis: {title: 'Salary', minValue: 0, maxValue: 10, format: '$#'},
      vAxis: {title: 'Percent above/below average', minValue: -1, maxValue: 1, format: '#%'},
      legend: 'none'
    };

    var chart = new google.visualization.ScatterChart(document.getElementById('chart_div'));

    chart.draw(dataTable, options);
  }
</script>

OK, so theoretically our PageController and our four Views should all be working at this point, we just need to add routes to our index.php for these pages. Much like our API routes, we'll specify a URL that includes variables in the URL string itself. Let's create a route for each of our three new Views:

// Page generator - class of functions for populating pages
$f3->route('GET /thanks/@name','PageGenerator->thanks');
$f3->route('GET /report/@name','PageGenerator->report');
$f3->route('GET /addResponse/@name','PageGenerator->addResponse');

Since these are normal web pages, we'll use the default GET method. Next we specify the path including the variable prefixed with the @ sign, and finally we tell F3 which class and function to map the route to (in the format of ClassName->functionName). Now we can hit those URLs with 'demo' as the name argument and we should see the pages populated with the details of the 'demo' survey object.

That's it - we now have a full working application including an API and a very decent user interface (it's even mobile friendly, thanks to Bootstrap).

Deployment Notes and 'Cloudiness'

This is already an insanely long write-up for this project, but I want to push through to make a couple important comments.

Before your F3 app is live in production, we need to make one more small change. By default, F3 will log error output directly to the browser. This is great for debugging, but unfortunately it can spit out sensitive information, like the username and password for our MongoDB. To disable this level of error logging, simply modify the 'DEBUG' F3 variable to 0, like this:

$f3->set('DEBUG',0);

Simple.

Deployment with OpenShift is trivial, as you're really doing continuous delivery - every time you push your code, OpenShift redeploys your environment with the new code. If you haven't been doing this along the way, you just need 3 commands to deploy your code:

# Execute this from within your OpenShift project directory
git add -A # add all changes to the git index
git commit -m 'add your commit message'
git push

Done. You can now hit the app by browsing to your OpenShift-generated DNS name. Adding your own custom DNS name is easy, and I'll cover that in a separate post.

As for cloudiness - the application might not be that complex and my object-oriented skills are garbage, but from an infrastructure/hosting perspective, it's the Holy Grail. Here's why:

The app doesn't care about DNS names. You can hit the app at http://shareyoursalary.alexdglover.com or http://shareyoursalary-alexdglover.rhcloud.com, or even http://www.shareyoursalary.gq/ and the app works perfectly. You can deploy it anywhere, change DNS names whenever, you can give customers/users their own custom 'vanity' URLs, and the app will give consistent results. It's extremely flexible and portable.
There are no sessions to worry about, making load-balancing extremely easy.
Since all of the app functionality is API based, we can scale the application horizontally just by adding more PHP instances behind the load balancer.
By using a NoSQL database (MongoDB), the database layer can also be scaled horizontally.
All of the code is deployment agnostic - since we're using environment variables for the database connection details, this same source can be deployed to development, QA, and production environments without any changes.
There are no high-cost or proprietary application components. You can run this app in any cloud hosting environment, on a number of operating systems, using off-the-shelf images/services in many cases.
The deployment is as easy as putting the contents of our project directory in the document root of the Apache/PHP servers. There's no configuration necessary.There is no DB schema to set up as part of the app deployment - the application will create the collection (read: table) on the fly and start inserting records.

In short, it should be very easy for anybody with minimal cloud automation training to deploy this application in a highly available and scalable model. Specifically, the app can handle instance failures in any tier and can scale horizontally to support hundreds of thousands of concurrent users without human intervention or configuration. With all of the advancements in cloud and automation, this should be the de facto standard for all but the largest and most complex apps.

As always, I hope this post has been helpful and let me know if you have any questions.

Share Your Salary App Part 2

2015-01-29T15:45:59+00:00

In Part 1, I covered the core classes for the Share Your Salary application. These classes covered my Model and Controllers in my quasi-MVC application. Now we're ready to expose those controller functions via a RESTful API.

For this project, I used the Fat Free Framework for both API routes and web page templating. To start, I simply downloaded and unzipped the F3 package and added the contents to my shareyoursalary app directory. Next, I created a "classes" folder and added the PHP class files I created in Part 1. At this point, my folder structure looked like this:

shareyoursalary/
- classes/
  - database.php
  - survey.php
- config.ini
- lib/
- .htaccess
- index.php
- readme.md
- ui/
  - css/
  - images/
  - layout.htm
  - userref.htm
  - welcome.htm

Now to create the API functionality, I created routes in F3. F3 makes it really easy to make HTTP method-specific routes that then map to pages, classes, or specific functions. In my case, I wanted to create routes for the various functions I created in the Survey class. To do this, we'll need to edit the index.php file in the root of the F3 directory. I left the existing code as-is, and added the following lines:

$f3->route('GET /api/v1/survey/getByName/@name','Survey->getByName');
$f3->route('POST /api/v1/survey/addResponse/@name','Survey->addResponse');
$f3->route('POST /api/v1/survey/addSurvey','Survey->addSurvey');

As you can see, the route syntax is:
'METHOD /URL/PATH/@parameter', 'Class->functionName'.

RESTful APIs should use certain methods based on what you're doing to the object (e.g. if you're reading an object, use GET, if you're creating an object, use POST). Here's a quick breakdown for reference:

GET - Reading an object
POST - Creating a new object (or overwriting an entire object)
PUT - Updating attributes of an existing object (this should be an idempotent operation)
DELETE - I'll give you three guesses.

There are other methods, but these are the four main methods you'll encounter with APIs.

Back to the F3 routes - you'll notice the @parameters in the URL. This tells F3 to pick up this portion of the URL and set it as a variable. You can then reference these parameters/variables in templates:

echo $f3->get('PARAMS.parameterName');

or you can use them inside of your classes/functions:

// Remember the route from index.php:
// $f3->route('GET /api/v1/survey/getByName/@name','Survey->getByName');
public function getByName($f3,$args) {
...
 $URLName = $this->convertNameToURLName($args['name']);
...

OK, so now I can hit http://shareyoursalary-alexdglover.rhcloud.com/api/v1/survey/getByName/test in a browser, and I'll get some JSON output in response (unless 'test' doesn't exist, in which case I'll get an HTTP 200 OK response but no output).

What about the other two routes? If you try to hit them in a browser, you'll get a nasty HTTP 405 Error - Method Not Allowed. Remember, a browser by default is GETting web pages, and F3 enforces the HTTP method you've assigned. So how are we going to test our API routes? We could install some browser plugin or execute cURL commands against it... but let's do something friendly for our API consumers, let's make it easy for people to explore the API. Enter Swagger UI.

I already gave an overview of Swagger UI in Part 1, now for the implementation details. To start, I downloaded the Swagger UI package, unzipped it the root of the project directory and renamed the directory 'swagger' leaving our new file structure like this:

shareyoursalary/
- classes/
  - database.php
  - survey.php
- config.ini
- lib/
- .htaccess
- index.php
- swagger/
  - bin/
  - dist/
  - lib/
  - src/
  - ...
- readme.md
- ui/
  - css/
  - images/
  - layout.htm
  - userref.htm
  - welcome.htm

Since swagger won't be managed by F3's template engine, I kept it outside of the ui directory. To make the Swagger web pages available, I add a new route in the F3 index.php file:

route('GET /swagger/',
  function($f3) {
    echo F3::serve('/swagger/dist/index.html');
  }
);

Now we can browse to /swagger/dist/index.html and we should see the Swagger UI page with the PetStore example. We're off to a good start.

Next, we need to modify /swagger/dist/index.html to refer to our new API JSON document (which we haven't created yet). On line 31, we need to provide the full URL path to the JSON file:

url = window.location.protocol + "//" + window.location.host + "/swagger/dist/shareyoursalary.json";

Note that this URL variable will work regardless of what DNS name the app is using (more on this later). Alright, now we'll create the shareyoursalary.json file to describe the API. Let's take a look at a few snippets and then break each one down:

{
"swagger": "2.0",
"info": {
"version": "0.0.1",
"title": "Share Your Salary"
},
"basePath": "/api/v1",
...

Pretty easy right? Just providing some version numbers, a title, and a base path. The base path is the most important piece here - this will be pre-pended to all of the other API URLs/path. Next, let's take a look at the first path:

"paths": {
  "/survey/getByName/{name}": {
    "get": {
      "description": "Gets `Survey` objects.\nRequired query param of **name** will be used as search criteria.\nShould only return one object.\nIf minimum entries haven't been met, responses will not be included in return data.\n",
      "parameters": [
        {
          "name": "name",
          "in": "path",
          "description": "name of survey",
          "required": true,
          "type": "string",
          "format": "string"
        }
      ],
      "responses": {
        "200": {
          "description": "Successful response",
          "schema": {
            "title": "SurveyJSON",
            "type": "array",
            "items": {
              "title": "Survey",
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "currency": {
                  "type": "string"
                },
                "period": {
                  "type": "string"
                },
                "minEntries": {
                  "type": "number"
                }
              }
            }
          }
        }
      }
    }
  }
}
...

Each path is identified by a key, which is the actual URL/path ("/survey/getByName/{name}" in this example). Each URL can have multiple functions based on what HTTP method is used, so we'll provide a description, parameters, and a response definition for each method. In this example, we are only dealing with GET. The parameter attributes are fairly straightforward except for the "in" attribute - the "in" attribute defines how the parameters are passed to this API. APIs can have parameter values passed in one of two ways:

As part of the URL path (e.g. /object/function/value)
As a query parameter (e.g. /object/function?parameter=value)

When passing a path parameter, we also need to add it to the routes inside of {curly braces}, like /survey/getByName/{name}. Finally, we define an expected response, which is straightforward. My entire "shareyoursalary.json" file is available on GitHub if you would like more example code. Now that we completed our JSON document describing our API, we can simply browse to the Swagger UI page and see all of the documentation as well as a working API client. Take a minute to check it out and fire off some API requests: http://shareyoursalary-alexdglover.rhcloud.com/swagger/dist/index.html.

Let's switch gears for a second, move away from the technical how-to and go over a few API philosophies and best practices.

Include a version in your API paths (you'll notice I used /api/v1/ in my base path). If you ever need to change the API in some way that's not backward compatible, you'll want to create a separate API route for the new version. This allows you to host two APIs, giving your downstream users the chance to re-write their code and switch to your new API before you kill the previous version.
Use the built-in HTTP methods and responses appropriately. This will make your API easier for you and your users to consume and debug. For example, throw an HTTP 405 error if someone sends a POST request to an API that only allows GET. If you threw an HTTP 500 error instead, people will wonder if there's a runtime error on the server rather than a mistake on their end. On that note, use HTTP 500 sparingly - this should only be thrown if you can't catch/handle an error appropriately, or it should be accompanied with some useful error text.
Make your API fun and easy to use. You want to encourage exploration and experimentation so other developers will build awesome applications on top of your already awesome API. You can accomplish this by using tools like Swagger and adding related calls to your API responses (i.e. if someone hits /invoice/123, send a "related objects" attribute in the response with routes to /invoice/122, /invoice/124, and /purchaseOrder/123 - that way the the user is encouraged to traverse or "walk" your API)

Awesome - now we have a working API service, our API is documented, and we can test it directly in case our AJAX-based UI has issues later.

In Part 3, we'll build a front-end (using jQuery, Bootstrap, F3, and Google Charts) that consumes our API service, set up DNS names, and review the application overall.

Share Your Salary App Part 1

2015-01-23T15:30:51+00:00

In a previous project, I built "Who's that person in that thing?," a JavaScript web app that called The Movie DB's API to find common actors given two movies. That was fun, but I still wanted to build an API service myself and I wanted to experiment with some new tools. But first I needed a project idea as an excuse to use them.

Project Idea/Application Premise/The Excuse

People are funny about money, especially about their salary. With most people, it's a taboo to talk about it. But everyone's been curious at some point to know how they compare to their peers. Some people fear they are getting paid less to do the same work as others, and sometimes that fear is founded in fact. The "Share Your Salary" app is an anonymous salary survey tool to address these kinds of use cases.

Before we get into any of the tools or how-to, feel free to check out the application first or look over the source.

Demo GitHub Repo

New Tools

OpenShift

Well I started to write this section, turned out to be about a thousand words... so I moved it to it's own separate post. In short, OpenShift is a PaaS hosting service. OpenShift hosts the entire Share Your Salary app, including a MongoDB container, an HAProxy load balancer, and Apache/PHP containers in an auto-scaling group.

MongoDB

MongoDB is a highly scalable NoSQL database. MongoDB is a document database and uses JavaScript Object Notation (JSON) to represent those documents. I really like JSON, but PHP doesn't 'speak' JSON natively, which makes using MongoDB a little less intuitive. I wrote a separate MongoDB and PHP Primer post to go over some of the basics.

I used MongoDB as the backend for the "Share Your Salary" app.

Fat Free Framework

Fat Free Framework (or F3 for short) is a tiny PHP framework (it's only ~65 KB in total) that provides URL routing, caching, page templating, and built-in database support for MySQL, SQLite, MSSQL/Sybase, PostgreSQL, MongoDB and F3's proprietary flat-file DB called Jig.

I used F3 for both templating web front-end as well as the API.

Swagger UI

Swagger UI is a must if you are building a RESTful API service - it acts as both a documentation method and a test bed. As the developer, you complete a JSON file that describes your API routes, methods, parameters, etc. Swagger UI transforms that JSON into a web page and API client that uses JavaScript to send HTTP GET/POST/PUT/DELETE requests to your API service and displays the results. Check out the Swagger Petstore Demo.

Retrospective

I would normally title this section "How-to" or something along those lines, but trying to remember all of the steps I went through after the fact is difficult. I tend to get caught up in a project and document afterwards (shame on me).

That said, I'll try to break down the highlights and lessons learned from this project.

The 'Database' Class

A quick disclaimer - I have a hard time thinking in object-oriented terms, I generally write functional programs. I tried to push myself to write more object-oriented with some simple classes to represent objects but I'm sure I botched it. Any criticisms or corrections are welcome.

Now, with any good OO program, you'll want to create a database class that can be instantiated by other classes. This allows you to write the basic database code once and re-use it elsewhere, instead of having a separate set of database connection code in every class that needs it.

Let's go over the code with some extra comments:

<?php

// Here we are defining a constant variable, OPENSHIFT_DB with the name
// of the MongoDB database we'll be connecting to, 'shareyoursalary'
define("OPENSHIFT_DB", "shareyoursalary");

class Database{

  // Here we define the constructor that will be called each time the
  // Database class is instantiated. In this case, we're just going
  // to call the get_db_connection() function
  function __construct(){
    $this->get_db_connection();
  }

  // Since I used OpenShift for this project, all of the database
  // connection information was abstracted to environment variables.
  // Ultimately, we are constructing a MongoClient class object and
  // returning it to whatever called this function
  function get_db_connection() {
    $host = $_ENV["OPENSHIFT_MONGODB_DB_HOST"];
    $user = $_ENV["OPENSHIFT_MONGODB_DB_USERNAME"];
    $passwd = $_ENV["OPENSHIFT_MONGODB_DB_PASSWORD"];
    $port = $_ENV["OPENSHIFT_MONGODB_DB_PORT"];
    $uri = "mongodb://" . $user . ":" . $passwd . "@" . $host . ":" . $port;
    $mongo = new MongoClient($uri);
    return $mongo;
  }

  // Returns a database object based on a database name input parameter ($dbname)
  function get_database($dbname) {
    $conn = $this->get_db_connection();
    return $conn->$dbname;
  }

  // Returns a collection object based on a collection name input ($collection)
  function get_collection($collection) {
    $db = $this->get_database(OPENSHIFT_DB);
    return $db->$collection;
  }

}
?>

This class is pretty straightforward and does little in the way of error handling. Basically we're just connecting to the database and abstracting some basic MongoDB functions to make them available as class functions.

The 'Survey' Class

The Survey object is the only real object in this program from an object-oriented programming perspective. This object (the survey) is the core of the application, and so most of the relevant data exists as properties of this class.

Let's walk through the source with some comments:

<?php

class Survey {

// Class properties are defined here. These should probably be all marked as private varaiables

  var $surveyName; // Human-readable survey name
  var $URLName; // URL-friendly version of the surveyName property
  var $currency; // the currency that this survey is based on, e.g. USD
  var $period; // The period that the survey is based on, e.g. per hour or per year
  var $minentries; // The minimum number of entries needed before the results are published. This ensures anonymity
  private $db; // Property to hold the Database object

  // This is the constructor function. It instantiates the database class and sets it to the local db property
  public function __construct(){
    $this->db = new Database();
  }

  // Simple getter functions for each of the properties. These aren't actually used anywhere, but I created them as a standard practice
  public function get_name(){
    return $this->surveyName;
  }

  public function get_url_name(){
    return $this->URLName;
  }

  public function get_currency(){
      return $this->currency;
  }

  public function get_period(){
    return $this->period;
  }

  public function get_minentries(){
    return $this->minentries;
  }

  // This function will take the human-readable name as an input, then convert and return the URL-friendly version
  private function convertNameToURLName($string) {
    //Lower case everything
    $string = strtolower($string);
    //Make alphanumeric (removes all other characters)
    $string = preg_replace("/[^a-z0-9_\s-]/", "", $string);
    //Clean up multiple dashes or whitespaces
    $string = preg_replace("/[\s-]+/", " ", $string);
    //Convert whitespaces and underscore to dash
    $string = preg_replace("/[\s_]/", "-", $string);
    return $string;
  }


  // This function finds a given survey in the database based on the surveyName
  // passed as an input parameter. The surveyName will be sent as part of
  // the $args parameter (sent as part of a URL path parameter via F3).
  // If found, the survey will be echo'd in JSON format (this
  // allows it to be consumed as an API)
  public function getByName($f3,$args) {
    $surveys = $this->db->get_collection('surveys');

    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($args['name']);

    $query = array('URLName' => $URLName);
    $cursor = $surveys->find($query);
    foreach ($cursor as $doc){
      if(count($doc['responses']) < $doc['minEntries'])
      {
        for($i=0; $i < (count($doc['responses'])); $i++)
        {
          $doc['responses'][$i] = '0';
        }
        echo json_encode($doc);
      }
      else {
        echo json_encode($doc);
      }
    }
  }

  // The addSurvey function takes HTTP parameters (the $_REQUEST variables) as
  // inputs and creates the new survey entry in the database. Once created, the
  // function attempts to find the survey and echo it in JSON format (this allows it to be consumed as an API)
  public function addSurvey($f3,$args) {
    // Get surveys collection    
    $surveys  = $this->db->get_collection('surveys');
    $surveyName   = $_REQUEST['surveyName'];
    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($surveyName);
    $currency  = $_REQUEST['currency'];
    $period    = $_REQUEST['period'];
    $minEntries   = $_REQUEST['minEntries'];

    // Insert new data sent via API call
    $surveys->insert(array('name' => $surveyName, 'URLName' => $URLName, 'currency' => $currency, 'period'=>$period, 'minEntries'=>$minEntries, 'responses'=>(array())));
    // Build a query object, basically a single item array with the name of the new survey input value
    $query = array('URLName' => $URLName);
    // Attempt to find the survey we just created
    $cursor = $surveys->find($query);
    // For each object that matches the query, echo the data as JSON
    foreach ($cursor as $doc){
      echo json_encode($doc);
    }
  }

  // The addResponse function takes a survey name (as a URL parameter passed via F3)
  // and a response input (via HTTP query parameters, the $_REQUEST variable below).
  // It uses the survey name to find the survey in the database and then updates the
  // embedded response array with the response value sent. Once updated, the function
  // returns the newly updated survey in JSON format
  public function addResponse($f3,$args) {
    $surveys = $this->db->get_collection('surveys');
    $response = $_REQUEST["response"];

    // Convert name to URL friendly name
    $URLName = $this->convertNameToURLName($args['name']);

    $surveys->update(array('URLName' => $URLName),array('$push' => array('responses' => $response)));
    $query = array('URLName' => $URLName);
    $cursor = $surveys->find($query);
    foreach ($cursor as $doc){
        echo json_encode($doc);
    }
  }
}
?>

Looking back, this Survey class contains aspects of both a model and a controller. In a proper MVC project, this probably should have been split into two classes, a 'Survey' object (with just the basic properties and getters/setters) and a 'SurveyController' (with the addSurvey and addResponse).

These classes might seem a bit abstract right now, but it'll make more sense once I explain the API routes in F3.

At 1400+ words, this is a good place to take a break. Check out Part 2 for the PageGenerator class, the F3 routes, and a how-to for Swagger UI.

MongoDB and PHP Primer

2015-01-08T15:30:24+00:00

I recently used MongoDB in the Share Your Salary app and had a few hiccups as I tried to blunder my way through learning how to use it. MongoDB, like many NoSQL document databases, uses JavaScript Object Notation or JSON to represent the objects/documents. JSON is great, especially when you're using JavaScript in conjunction. PHP, however, doesn't 'speak' JSON natively; it uses arrays instead, and this makes for some less than intuitive interactions. This post will go over some of the basic CRUD operations using PHP against a MongoDB.

Connecting

Connecting is pretty straight-forward:

$host = your.mongodb.host.com;
$user = username;
$passwd = Sup3rS3cr3tPassword;
$port = 27017; // 27017 is the default port
$uri = "mongodb://" . $user . ":" . $passwd . "@" . $host . ":" . $port;
$mongo = new Mongo($uri);

If you're using OpenShift, you can simply use the relevant environment variables:

$host = $_ENV["OPENSHIFT_MONGODB_DB_HOST"];
$user = $_ENV["OPENSHIFT_MONGODB_DB_USERNAME"];
$passwd = $_ENV["OPENSHIFT_MONGODB_DB_PASSWORD"];
$port = $_ENV["OPENSHIFT_MONGODB_DB_PORT"];
$uri = "mongodb://" . $user . ":" . $passwd . "@" . $host . ":" . $port;
$mongo = new MongoClient($uri);

Reading a 'table'

In NoSQL document databases, an individual document is analogous to a record in a traditional relational SQL database. Similarly, a collection is analogous to a table.

Before we can fetch a collection, we have to specify which database we want to work with. After all, one MongoDB instance can host multiple databases.

database_name;
$collection = $db->selectCollection("collection_name");

Or if you want to save some typing:

selectCollection("database_name", "collection_name");

Now that you have a collection object, you can query it, iterate through it, etc.

Querying documents

When querying MongoDB, a cursor object is returned. This cursor must be iterated to get to the actual data in the records returned.

To get all documents in a collection, just call the find() method:

$cursor = $mongo->database_name->collection_name->find();

This is similar to a "SELECT * FROM table_name" query in a relational SQL database.

To query with specific criteria, you'll need to construct an array of the key-value pair you want to query on. Let's assume we had a few JSON documents with the following structure:

{ "name": "alex", "age": 28, "computers": ["laptop","desktop","vm"] },
{ "name": "whoever", "age": 35, "computers": ["laptop","desktop"] }

To search the "name" keys for the value "alex" you'll need to create an array with that key-value pair, then execute the find method with that query.

$query = array('name' => 'alex');
$cursor = $collection->find($query);

Or, more succinctly:

cursor = $collection->find(array('name' => 'alex'));

This is similar to a "SELECT * FROM table_name where name='alex' " query in a relational SQL database.

But what if you wanted something less specific, like all documents where the 'name' value started with 'a' ? In a relational SQL query, you would used the LIKE operator and wildcards (%). With MongoDB, you just use regular expressions in your query. So, to get all documents where the 'name' value started with 'a' you would execute:

$query = array('name' => array('$regex' => new MongoRegex("/^a.*/")));
$cursor = $collection->find($query);

If you're not familiar with regular expressions, spend some time at Regex101. Super useful, pretty easy to learn.

If you want to return a single result array instead of a cursor, use the findOne() method.

Inserting a new document

Inserting a new row/document is really easy. Since the collection (and all of MongoDB for that matter) doesn't have any schema or structure to enforce, you can insert any array into any collection, so long as it's a valid array.

$collection->insert(array('name' => 'bob','age' => 40 ));
$collection->insert(array('someUnrelatedKey' => 'someUnrelatedValue'));

You can also inserted nested arrays (which will become nested objects in JSON):

$collection->insert(array('name' => 'bob','age' => 40,
  'computers' =>(array('laptop','desktop')) ));

Updating an existing document

Updating a document is where MongoDB shows it's flexibility as a NoSQL database. It's also the trickiest of the CRUD operations. The basic structure with psuedo-code is as follows:

$collection->update(search_criteria, 
  array('$set' => array("key => "value") ));

Here's an example with legitimate values:

$collection->update(array("name" => "alex"), 
  array('$set' => array("name" => "bob")));

This query is equivalent to "UPDATE collection SET name='bob' where name='alex';" in traditional SQL.

Notice the $set argument - without this, MongoDB will REPLACE the entire document with the array you passed instead of trying to update a particular element. There are several update operators for MongoDB that can be very helpful. For example, let's assume we had the following document object:

{ "name": "bob", "age": 35, "computers": ["laptop","desktop"] }

Notice that we have a nested array of "computers" with two values. If we wanted to add another element to the "computers" array, we'd have to fetch the entire document, strip out the nested array object, add our new element, then use the $set operator to update the array in the document.

Fortunately, MongoDB provides a special update operator for this use case, $push. Let's pretend we want to update bob's record in the database and give him a VM.

$collection->update(array('name' => 'bob'),
  array('$push' => array('computers' => 'vm')));

Awesome.

This was a pretty elementary tutorial on MongoDB, but I'm hoping it either helped one of you get started or inspired someone to use MongoDB in their project. Everything I covered in this post and much more can be found by perusing the Mongo Book at php.net and MongoDB's Reference.

Introduction to OpenShift

2015-01-03T23:33:25+00:00

OpenShift is a Platform-as-a-Service offering provided by Red Hat. It's built on top of Linux containers (which they refer to as 'gears') hosted in Amazon's Amazon Web Services (AWS). They offer many different platforms such as PHP/Apache, JBoss, Tomcat, NodeJS, Python, Ruby-on-Rails, and many more. OpenShift refers to these pre-built platforms as 'cartridges.'

Beyond the platforms they offer, OpenShift also takes care of DNS (you are given a *.rhcloud.com subdomain), load balancing via HAProxy, and auto-scaling for your application. Much like Heroku, code deployment is as easy as executing a git push. All of these features, including 3 containers, are all free of charge.

Red Hat has done a good job making OpenShift "cloudy" - for example, database connection strings are handled by environment variables. This is cloud-friendly because:

Your source code can now reference the environment variables instead of storing credentials in your source (preventing you from using public repositories) or having a separate private deployment repository with credentials that you have to manage.
Your source code can now be deployed instantly in another OpenShift environment, no need to change credentials in the code.
You don't have to worry about passing credentials to new instances during auto-scaling, OpenShift takes care of it for you.
Your application and operations folks can deploy and manage your application without them having access to the database credentials.

This is just one example but I think it does a good job illustrating the foresight Red Hat had when they built OpenShift.

If you're a developer and don't want to worry about the infrastructure details yet, OpenShift is really an amazing service. You can set up an application in just a few minutes, I'll show you how. Let's get started.

Initial Setup

Sign up for an account at http://www.openshift.com and verify your account once you receive the confirmation email.
When you verify your email, you'll get redirected to a welcome page that invites you to "Create your first application now." Go ahead and click on it.
Find the type of application you want to deploy, either by browsing the available cartridges or searching. Keep in mind, you can add additional cartridges (like Jenkins or MySQL) later, so don't freak out if you can't find an entire LAMP stack or something. For this example, I'll choose the Python 3.3 cartridge.
On the application configuration page, you'll be prompted to provide an application name (which will be pre-populated with the cartridge name) and a namespace name. Heads up - the namespace you set now will be used for ALL future applications as well. Think of it like a subdomain - use something appropriate like your user name or your organization's name. Fear not, you can create custom DNS CNAME records later if you need a sexier URL.
If you already have code that you want to deploy in a git repository, you can provide a git URL and Openshift will automatically clone a copy and deploy it to your container. Note - if your code relies on OpenShift environment variables for a database connection string, don't use this feature. Wait until after you add all cartridges before deploying your code, otherwise those environment variables won't be set. If you're using a free account, you will only be able use the 'small' sized gears, but you can take advantage of auto-scaling. Finally, choose which AWS region you want to deploy to and click "Create application."
There are a number of configuration options presented in the application summary page:
You can set the minimum and maximum number of containers as well as container size in your auto-scaling group by clicking the "1-3" under the "Scales" column header. Clicking on the gear size will take you to the same page.
If you upgraded beyond the free account, you can set how much local storage is presented to your container by clicking on the "1 GB" under the "Storage" column header.
To see HAProxy load balancer statistics, click the arrow icon on the Web Load Balancer row.
You can add additional cartridges by clicking on the "Add X" links at the bottom of the page. These cartridges will be bound to your application and restarted in conjunction with your application. If you add a database cartridge, connection string variables will be set as environment variables for your application to leverage. Here's a list of relevant OpenShift environment variables.
You can add DNS names by clicking "change" next to your *.rhcloud.com DNS name.

That's it for deployment. At this point you'll want to install the OpenShift CLI tools to deploy new code, check application logs, and manage application components.

Deployment Overview

Once you have the OpenShift CLI tools installed and configured, deploying code is extremely easy. To start, clone the git repository to your local filesystem. You can find the git URL in your application configuration screen labelled "Source Code":

Clone the repository:

git clone ssh://<random_hash>@appname-namespace.rhcloud.com/~/git/appname.git/

Change to your newly created app directory:

cd appname/

Add your files, make changes, hack away to your heart's content in this directory. Once you're ready to re-deploy, add the files to the git index, make a commit, and push the latest source:

git add .
git commit -m 'this is my commit message'
git push

When you push new code, all application cartridges will be restarted and your new application should be available in a few seconds. If you need to add additional startup commands for your application, you can add additional action hooks.

That's it - you should now be able to create and deploy an application with OpenShift with ease.

Are you using OpenShift, or some other PAAS offering? What are your thoughts?

Download All Files From A Site With wget

2014-12-02T15:00:03+00:00

I recently got a membership to a site hosting a boatload of private label rights (PLR) material (IDPLR.com). 99% of PLR items are scams, garbage, or are outdated, but if you have the time or tools to dig through it you can find some gems. (Just so we're clear, I'm not running any scams, I was just looking for some content for another project). Anyway, I got access to the site that had thousands of zip files but obviously didn't want to download them one by one. Administrator wouldn't give me any way to download the files in bulk either, so I was left with no choice but to automate...

This is one of those projects that can be done a dozen different ways. There's probably a real slick way to do this with one line of code, but this method worked for me.

Step 1 - Identify your targets

This was the most challenging part for me, until I got lucky and found a blog post on how to grab EVERY link on a site using wget. I can't take any credit for this script, so I'll just share my slightly modified version:

MYSITE='http://www.idplr.com'
wget -nv -r --spider $MYSITE 2>&1 | egrep ' URL:' | awk '{print $3}' | sed "s@URL:${MYSITE}@@g" >> links.txt

This will create a file, links.txt, that contains EVERY link found on the site you specified for your "MYSITE" variable. Very powerful one-liner for sure. Keep in mind it may take a very long time for this command to complete (about an hour in my case).

We don't want to download every file on the site, so we need to whittle down our list. In my case, all of the files I wanted to download were conveniently in a directory labeled "downloads" (albeit in several different child directories) so I used the following command to create a new file (downloads.txt) that contained only the links to the "downloads" directory.

less links.txt  | grep downloads/ >> downloads.txt

The downloads.txt file looks like this:

...
/members/protect/new-rewrite?f=11&url=/downloads/ebooks/gold/AffiliateFitnessCraze_rr.7594.zip&host=www.idplr.com&ssl=off
/members/protect/new-rewrite?f=11&url=/downloads/ebooks/gold/AzonHairLossEssentials_rr.7592.zip&host=www.idplr.com&ssl=off
/members/protect/new-rewrite?f=11&url=/downloads/ebooks/gold/AzonHalloweenEssential_rr.7591.zip&host=www.idplr.com&ssl=off
...

You'll see that the site scraper pulled the URL of the redirect that is protecting the members-only files. We don't want the strings appended and pre-pended to the path, we just want the /downloads.../filename.zip. So let's use 'awk' to strip out all of that garbage and give us just the path to the files.

awk -F'&url=' '{print $2}' downloads.txt | awk -F'&host' '{print $1}' >> cleanLinks.txt

Let's break down what the awk command does in this case. The -F flag allows you to set a delimiter, in this case '&url=', to break up the string. Since that string only appears once per line, the line is broken into two strings - all of the characters leading up to the delimiter, and all of the characters after the delimiter, not including the delimiter. Then we use '{print $2}' to print that second chunk of the string. So at this point we've stripped the string down to:

"/downloads/dirname/dirname/filename.zip.&host=www.idplr.com&ssl=off."

Well we don't any of the garbage after the file name, so we'll pipe the output from the first awk command into a second awk command. Again, we specify a delimiter (-F '&host') to split the string into two pieces, then print only the first half of the string ('{print $1}' above). The output now will be "/downloads/dirname/dirname/filename.zip" - perfect! Finally, by using the ">>" redirection, the output from these commands will be written to a new file, cleanLinks.txt. Target acquired.

Step 2 - Scripting the Download with Login Parameters

Just for a test, I tried to run 'wget' against one of my targets, and ended up with an HTML file. That's because the site was redirecting me to the log in screen, and wget followed the redirect and downloaded that page instead. Same thing would happen in your browser; if you try to browse to that link without authenticating first, you get re-routed to the log in screen. No problem - we just need to figure out how the redirect and log in screens are working. I'm using the built-in console in Chrome for this, but there are many tools out there for looking at HTTP requests and responses. If you are following along word for word, pull up Chrome, punch in the link for a protected file and let the browser get redirected to the log in screen. Then press F12, or right click anywhere on the site and choose "Inspect Element." The console should pop up on the bottom of the browser window. Click the Network tab, and the record button (gray circle at the bottom, will turn red when clicked). For example:

The console will now record all HTTP requests and their responses, even as you traverse to different pages (if you didn't click the "record" button, it only shows you current requests). Now, log in using your credentials and watch all of the requests and responses fly past. The file should also started downloading - if not, go back, because the next part will be worthless if you didn't authenticate properly and get to the file you expected to. Find the request on the left of the console labelled 'login' (this could obviously be different if you're downloading from a different site) and look at the request headers. Under Form Data, click "view source".

The key information is highlighted in the red boxes. The request URL is the URL we sent our HTTP request to. The form data is the username and password we entered, plus some hidden and generated fields that were in the form (check the w3schools page if you need more information on forms in HTML). Now, to verify it works, try this wget command:

wget http://www.idplr.com/members/login -O <insert file name here> \
  --post-data="amember_login=<your_username>&amember_pass=<your_password>&login_attempt_id=1384717099&amember_redirect_url=<link_to_the_file>

So for example, if I wanted to download "SlipstreamMarketing.2196.zip", my command would like this:

wget http://www.idplr.com/members/login -O SlipstreamMarketing.2196.zip \
  --post-data="amember_login=alexdglover&amember_pass=E1zxxxxxxxxxxx&login_attempt_id=1384717099&amember_redirect_url=http://www.idplr.com/downloads/ebooks/free/SlipstreamMarketing.2196.zip"

Now, if you've gotten this far and everything is working, the rest is cake.

Step 3 - Adding the Loop

At this point, we have a text file that has a path to each file we want to download on each line (cleanLinks.txt), and a way to authenticate and download that file in one command (see Step 2). We just need to write a bash script that executes that command for each line in the file. Create a file with your favorite text editor, and enter the following code:

# execute the following code block for each line in the text file
while read line; do
        # Create a new variable, filename, to hold the actual filename instead of the full path (name.zip, instead of /path/to/file/name.zip)
        filename=$(basename $line)
        # Echo the filename out to the terminal for debugging purposes
        echo $filename
        # Echo out the path that wget will attempt to use to download a file
        echo "Attempting to download file at http://www.idplr.com$line"
        # Execute our wget one-liner, but this time with the filename and path to the file as variables ($filename and $line)
        wget http://www.idplr.com/members/login -O $filename --post-data="amember_login=alexdglover&amember_pass=E1zxxxxxxxxxx4&login_attempt_id=1384717099&amember_redirect_url=http://www.idplr.com$line"
# End of the while loop, and setting the text file (cleanLinks.txt) that the loop should read
done < cleanLinks.txt

Don't forget to save your new script file, and execute "chmod u+x scriptname.sh" to make it executable. Be advised - the script will download files to the whatever directory you execute it from (if the script is in /home/alex/script.sh, but my current working directory is /home/alex/downloads/ and I execute the script, all of the downloads will end up in /home/alex/downloads/). Once you're in the right directory, just execute the script, sit back, and relax.

Who's that person in that thing? App

2014-07-28T13:00:44+00:00

I've been looking for an API project for awhile now, either to write an API service or a client/app that consumes one. Then I finally thought of a use case - an app that would allow you to find an actor based on two movies they've been in, using only JavaScript and APIs. It took me awhile to find a movie API service that was free and suited my purposes, then I found TheMovieDB (which has a pretty sweet API actually and is free to use). Check out the demo and grab the source from GitHub if you want to do something similar.

Demo GitHub Repo

HTML Resume Template

2014-07-21T13:30:40+00:00

I'll keep this short since it's not a proper how-to post or lengthy tutorial. I recently discovered a resurgence in the concept of HTML resumes and was thoroughly impressed. As always, I recklessly decided I should take a stab at it myself even though I knew I was out of my depth. Ended up with a half-decent HTML5/responsive/mobile-friendly resume using Bootstrap 2.3 and wanted to share it with the masses. Check out the demo and if you want to build on it, grab the source. Enjoy.

Demo GitHub Repo

Chef and iptables

2014-07-14T13:30:23+00:00

Like many DevOps professionals before me, I needed a way to manage iptables programmatically through Chef. Before I realized that Opscode had written a cookbook for just this, I had already spent some time building my own. The cookbook written by Opscode is very clever and allows you to apply a set of iptables rules with a single line in another recipe. The only problem is that it is predicated on you creating 'templates' of iptables policies you want to apply in advance, and you can only apply those templates. You can't just supply a port number or part of a rule. Additionally, their cookbook wasn't compatible with Vagrant because of the ruby binary wasn't in the paths used in the cookbook.

Short version - I wrote my own iptables cookbook and augmented the Opscode iptables cookbook, and I wanted to share.

Changes to the Opscode Cookbook

Basically, the cookbook functionality is unchanged save for two parts. First, in addition to the previous method for invoking iptables, you can now invoke it by setting the ['iptables']['roles'] attribute to one or more roles (comma delimited). There is a new 'default.rb' attribute file and the default recipe has been modified slighly to support this. Second, the default recipe has been modified to detect the path to the ruby binary so as to be compatible with Vagrant boxes instead of being hardcoded.

You can check out all of the code on Github.

My iptables Cookbook

My requirements were a little less generic, and so my iptables cookbook is a little more isolated than the Opscode version. Specifically, I didn't want to affect any existing iptables rules or chains, so my cookbook injects new chains that should be specific for the servers application (hence App-INPUT and App-OUTPUT chains). Unfortunately, today my cookbook only supports ports to be specified via attributes and only works on Enterprise Linux OS's. Having seen and played with Opscode version, I'll have to refactor mine to support Debian Linux versions as well as non-attribute driven usage.

To see the full cookbook or to clone, check out the repo on Github. Otherwise the bulk of the logic is in the default recipe, which you can check out here:

# Cookbook Name:  inline-iptables
# Recipe:         default
# Author:         Alex D Glover (alex@alexdglover.com)
# Description:    A recipe to manage iptables without impacting existing
#   iptables rules or chains, and allowing individual
#   inbound/outbound ports as well as port ranges to be
#   managed with programmatic ease.
# Dependencies:   attributes/default.rb
# Usage:          Set the ["inline-iptables"]["listen_ports"] and/or
#   ["inline-iptables"]["outbound_ports"] attributes at the
#   node or role level


# Grab the comma separated list of ports that need to be open.
# Keep in mind these attribute could be set by a role, recipe, or node override
listen_ports      = node["inline-iptables"]["listen_ports"]
outbound_ports    = node["inline-iptables"]["outbound_ports"]

# Boolean variable to track if we a change has been made
iptables_modified = false

# Debug friendly logging
Chef::Log.info <<-EOS

Entering ondemand_base::iptables_manager {
  listen_ports        = #{listen_ports}
  outbound_ports      = #{outbound_ports}
}
EOS

# Read the current iptables data
iptables_content = File.read("/etc/sysconfig/iptables")

# If the list of ports is the empty string, do nothing
unless listen_ports == ""

  # We are creating a new iptables chain called App-INPUT; if it already
  # exists, don't create it again
  unless iptables_content.include?(":App-INPUT - [")
    execute "create new INPUT chain" do
      command "iptables -N App-INPUT;"
    end
    iptables_modified = true
  end

  # Break our port list into an array
  listen_ports_array = listen_ports.split(',')

  # For each port that needs to be opened, insert the corresponding rule
  # into our chain, but only if doesn't exist already
  listen_ports_array.each do |port|
    unless iptables_content.include?("-A App-INPUT -p tcp -m tcp --dport #{port} -j ACCEPT")
      execute "add input rule to iptables for port #{port}" do
        command "iptables -I App-INPUT -p tcp --dport #{port} -j ACCEPT"
      end
      iptables_modified = true
    end
  end

  # Connect our App-INPUT chain to the generic INPUT chain
  unless iptables_content.include?("-A INPUT -j App-INPUT")
    execute "connect App-INPUT to INPUT chain" do
      command "iptables -I INPUT -j App-INPUT"
    end
    iptables_modified = true
  end

  # Connect our App-INPUT chain to the generic FORWARD chain
  unless iptables_content.include?("-A FORWARD -j App-INPUT")
    execute "connect App-INPUT to FORWARD chain" do
      command "iptables -I FORWARD -j App-INPUT"
    end
    iptables_modified = true
  end

else
  Chef::Log.info "No listen ports specified to be opened"
end

# If the list of ports is the empty string, do nothing
unless outbound_ports == ""

  # We are creating a new iptables chain called App-OUTPUT; if it already
  # exists, don't create it again
  unless iptables_content.include?(":App-OUTPUT - [")
    execute "create new OUTPUT chain" do
      command "iptables -N App-OUTPUT;"
    end
    iptables_modified = true
  end

  # Break our port list into an array
  outbound_ports_array = outbound_ports.split(',')

  # For each port that needs to be opened, insert the corresponding rule
  # into our chain, but only if doesn't exist already
  outbound_ports_array.each do |port|
    execute "add input rule to iptables for port #{port}" do
      command "iptables -I App-OUTPUT -p tcp --dport #{port} -j ACCEPT"
      only_if {!iptables_content.include?("-A App-OUTPUT -p tcp -m tcp --dport #{port} -j ACCEPT")}
    end
    iptables_modified = true
  end

  # Connect our App-OUTPUT chain to the generic OUTPUT chain
  unless iptables_content.include?("-A OUTPUT -j App-OUTPUT")
    execute "connect App-OUTPUT to OUTPUT chain" do
      command "iptables -I OUTPUT -j App-OUTPUT"
    end
    iptables_modified = true
  end

else
Chef::Log.info "No outbound ports specified to be opened"
end

if iptables_modified

  execute "save updated iptables" do
    command "/etc/init.d/iptables save"
  end

  service "iptables" do
    action :restart
    ignore_failure true
  end
else
  Chef::Log.info "No changes made, not restarting iptables"
end

Would greatly appreciate any feedback/criticism from the Chef community. Thanks for reading.

Bootstrap Scrollspy Workaround

2014-06-16T13:00:10+00:00

I was recently working on a new HTML project when I came across the Scrollspy feature in Bootstrap's framework (v2.3). The feature looks really good on Bootstrap's pages, I figured I'd use it in my project as well. I immediately had issues with the links aligning the content to the viewport and issues with the scrollspy highlighting the right link based on what content was in the viewport. This was all supposed to be out of the box functionality... After doing some research (read: Googling), I came across a few bug tickets and workarounds, but no wholistic solutions to reproducing the functionality on Bootstrap's site. So I decided to build it manually.

Let's get started.

Preparation

If you want to jump right into the project, you can download the starter kit (it's just Boostrap 2.3, a modified Hero Unit example page, and a very slightly modified Bootstrap CSS).

If you want to start from a 100% clean version of Bootstrap 2.3, go download the source and extract it. Keep your top-level folder name whatever you want it to be, but rename the 'bootstrap' folder assets (we are matching the existing folder structure in the source code we are going to borrow from one of their examples. Feel free to build your folder structure however you see fit). Create a sibling folder to contain your HTML files and create an index.html inside. Finally, go download jQuery, place it in your assets/js folder, and rename it 'jquery.js' (for simplicity). Your folder structure should look like this:

topLevelFolder
├── assets/
│   ├── css/
│   │   ├── bootstrap.css
│   │   ├── bootstrap.min.css
│   ├── js/
│   │   ├── bootstrap.js
│   │   ├── bootstrap.min.js
│   │   ├── jquery.js
│   └── img/
│       ├── glyphicons-halflings.png
│       └── glyphicons-halflings-white.png
├── html/
│   ├── index.html

Perfect. Now browse to the Hero example and steal borrow their source code (just right click anywhere, view source, Ctrl+A Ctrl+C) and paste it into your index.html file. We have to make a few minor changes here as well. Scroll to the bottom of the code and replace this:

<!-- Le javascript ================================================== -->
    <!-- Placed at the end of the document so the pages load faster -->
    <script src="../assets/js/jquery.js"></script>
    <script src="../assets/js/bootstrap-transition.js"></script>
    <script src="../assets/js/bootstrap-alert.js"></script>
    <script src="../assets/js/bootstrap-modal.js"></script>
    <script src="../assets/js/bootstrap-dropdown.js"></script>
    <script src="../assets/js/bootstrap-scrollspy.js"></script>
    <script src="../assets/js/bootstrap-tab.js"></script>
    <script src="../assets/js/bootstrap-tooltip.js"></script>
    <script src="../assets/js/bootstrap-popover.js"></script>
    <script src="../assets/js/bootstrap-button.js"></script>
    <script src="../assets/js/bootstrap-collapse.js"></script>
    <script src="../assets/js/bootstrap-carousel.js"></script>
    <script src="../assets/js/bootstrap-typeahead.js"></script>

with this:

<!-- Le javascript ================================================== -->
    <!-- Placed at the end of the document so the pages load faster -->
    <script src="../assets/js/jquery.js"></script>
    <script src="../assets/js/bootstrap.js"></script>

For sanity testing, open index.html in your browser - it should look like the original and function without errors.

Now let's strip out the 3 sub-headings and replace them with a content area and a set of links in the sidebar.

Replace this:

      <div class="row">
        <div class="span4">
          <h2>Heading</h2>
          <p>Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui. </p>
          <p><a class="btn" href="#">View details &raquo;</a></p>
        </div>
        <div class="span4">
          <h2>Heading</h2>
          <p>Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui. </p>
          <p><a class="btn" href="#">View details &raquo;</a></p>
       </div>
        <div class="span4">
          <h2>Heading</h2>
          <p>Donec sed odio dui. Cras justo odio, dapibus ac facilisis in, egestas eget quam. Vestibulum id ligula porta felis euismod semper. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p>
          <p><a class="btn" href="#">View details &raquo;</a></p>
        </div>
      </div>

with this:

      <div class="row">
      <div class="span3 sidebar">
        <ul class="nav nav-list nav-tabs nav-stacked sidebarNav affix-top" style="top:80px; width:255px;">
        <li id="section1Nav"><a href="#section1">Section 1<i class="icon-chevron-right" style="float:right;"></i></a></li>
        <li id="section2Nav"><a href="#section2">Section 2<i class="icon-chevron-right" style="float:right;"></i></a></li>
        <li id="section3Nav"><a href="#section3">Section 3<i class="icon-chevron-right" style="float:right;"></i></a></li>
        <li id="section4Nav"><a href="#section4">Section 4<i class="icon-chevron-right" style="float:right;"></i></a></li>
      </ul>
      </div>

      <div class="span9">
      <section id="section1">
        <div class="page-header">
          <h1>section 1</h1>
        </div>
        <h2>Sub Header</h2>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
      </section>
      <section id="section2">
        <div class="page-header">
          <h1>section 2</h1>
        </div>
        <h2>Sub Header</h2>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
      </section>
      <section id="section3">
        <div class="page-header">
          <h1>section 3</h1>
        </div>
        <h2>Sub Header</h2>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
      </section>
      <section id="section4">
        <div class="page-header">
          <h1>section 4</h1>
        </div>
        <h2>Sub Header</h2>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
        <p>lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.</p>
      </section>
      </div>
      </div>

Open this up in your browser and verify everything is kosher. If so, we're all done with prep and can move on to the custom JavaScript.

The Offset Logic

At this point, you probably clicked the links and noticed that the viewport doesn't line up quite perfectly. This is because the height of the top nav bar isn't being taken into account. Let's fix that first, then reproduce the scrollspy effect.

To fix this issue, we need to 'intercept' the normal link function and replace it with our own. Once intercepted, we'll scroll to that element with an offset equal to the height of the header.

<script>
    // Here we set the offset value to whatever the header height is
    var offset = 60;
    // If a user clicks on any anchor element (a) within a list item tag (li) on our sidebar with the class sidebarNav (.sidebarNav), execute this function
    $('.sidebarNav li a').click(function(event) {
        // Prevent the normal scroll behavior
        event.preventDefault();
        // Grab the clicked element's href tag, scroll that element into view
        $($(this).attr('href'))[0].scrollIntoView();
        // Scroll 0 pixels horizontally and -60 (really negative offset) vertically
        scrollBy(0, -offset);      
    });
</script>

That's it. Add this block of javascript in your index.html just after your script source declaration for jQuery and boostrap.js. Test it in your browser, and experiment with changing the offset number and its effects.

The Scrollspy Effect

To create the Scrollspy effect, all we need to do is add the 'active' class to the li tags when the viewport is lined up with the corresponding section.

<script>
$(document).ready(function () {
  var menu = $('.sidebarNav');

  $(window).scroll(function () {
    var currPosition = $(this).scrollTop();
    var section1Position = $('#section1').offset().top - 100;
    var section2Position = $('#section2').offset().top - 100;
    var section3Position = $('#section3').offset().top - 100;
    // The offset for section 4 is larger because it is at the bottom of the page,
    // and if the offset was only 100, the window position would never 'enter' the
    // section's field
    var section4Position = $('#section4').offset().top - 150;

    // If the window is at or below the section 1 tag, then switch the
    // sidebar's class from affix-top to affix so it will move with the window
    if (currPosition >= section1Position) {
      menu.removeClass('affix-top').addClass('affix');

    }
    // If the window is NOT at or below the section 1 tag, then switch the
    // sidebar's class from affix to affix-top so it will position itself
    // below the Hero Unit
    else{
      menu.removeClass('affix').addClass('affix-top');
    }

    // If the window is at or below the section 1 tag, then add the active class
    // to the corresponding li tag and remove the active class from all other li
    // tags in the side bar. Apply the same logic to all of the other sections
    if (currPosition >= section1Position && currPosition <= section2Position)
    {
      $('#section1Nav').addClass('active').siblings().removeClass('active');
    }
    else if(currPosition >= section2Position && currPosition <= section3Position)
    {
      $('#section2Nav').addClass('active').siblings().removeClass('active');
    }
    else if(currPosition >= section3Position && currPosition <= section4Position)
    {
      $('#section3Nav').addClass('active').siblings().removeClass('active');
    }
    else if(currPosition >= section4Position)
    {
      $('#section4Nav').addClass('active').siblings().removeClass('active');
    }
  });
});
</script>

Again, you'll need to add this code block after the script source declarations for jQuery boostrap.js. That's it. Now you've got a dead-accurate version of the scrollspy, with complete control. Demo and source for the complete package are below.

Demo GitHub Repo

Enjoy.

Raspberry Pi Project 1: NAS and XBMC

2014-01-04T22:08:19+00:00

The Raspberry Pi makes a great HTPC or media PC, whatever you want to call it, because of its low cost, low power consumption, quiet, and small form factor. As I was getting ready to start my Raspberry Pi build, I realized that the Pi could take over a few other functions that my desktop does, including NAS, at a fraction of the cost (see my previous post about Powering a Pi and related utility savings). But could I modify a RaspXBMC build to also run Samba or some other network file sharing? Could XBMC be installed on Raspbian with less than a thousand calls to aptitude? Could XBMC and Samba run simultaneously on a Pi without making video playback unwatchable? That's what I hoped to find out.

For context, I started with a Raspberry Pi starter kit from Amazon. For storage, I am using a Fantom 2TB external hard drive. After some preliminary research, I decided to start with Raspbian and try to install XBMC on top. For this walk-through, I'll assume you've got your Raspberry Pi powered, connected to your router, and that you have gotten Raspbian installed. If you need help getting started, check out the Raspberry Pi Quick Start Guide.

Part 1 - NAS

Part 1.1 - NTFS-3G

If you don't plan to use NTFS as your filesystem for your Pi storage, you can skip this step!

If you do want to use NTFS, and you want to be able to read AND write to the drive, you need to install NTFS support:

sudo apt-get install ntfs-3g

If you're starting with a brand new external hard drive, you'll need to create the partition and filesystem.

pi@raspberrypi ~ $ sudo fdisk /dev/sda

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): 1
Command (m for help): t
Selected partition 1
Hex code (type L to list codes):7
Command (m for help): w
...
(disk is formatted)
...
pi@raspberrypi ~ $ mkfs.ntfs /dev/sda1

Part 1.2 - Automounting with fstab

In Raspbian (and every flavor of Linux I've worked with so far), all filesystems that should be mounted at boot are identified in the fstab file (/etc/fstab). The fstab file can be configured a million ways, so I'll just share my configuration and explain.

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p5  /boot           vfat    defaults          0       2
/dev/mmcblk0p6 /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

#/dev/sda1 - the 2TB Fantom Drive
/dev/sda1       /media/1TB      ntfs-3g defaults        0       0

The first four lines were created automatically by the Raspbian install, we'll leave those alone. We're only interested in the last line that starts with /dev/sda1, that's the line I've manually added to the configuration.

BUT WHAT DOES IT ALL MEAN??? The columns of the fstab file (courtesy of the ArchLinux wiki) are

# <file system> <dir> <type> <options> <dump> <pass>

<file system> - the partition or storage device to be mounted. In my case, “/dev/sda1”

<dir> - the mountpoint where <file system> is mounted to. In my case, “/media/1TB”

<type> - the file system type of the partition or storage device to be mounted. In my case, it is “ntfs-3g”

<options> - mount options for the filesystem. In my case, I chose “defaults” which includes mounting the filesystem as read-write with no umasks for permissions/access

<dump> - not needed in our case, just use 0 (zero)

<pass> - used by fsck to decide which order filesystems are to be checked. You can set this to 0 (zero)

So my configuration file says to mount /dev/sda1 on /media/1TB as an NTFS filesystem, using default mount options, and never run fsck on this drive.

Now that you have an example, implementing is pretty easy.

Make a backup of your fstab file

sudo cp /etc/fstab /var/fstab
Open fstab in your favorite text editor

sudo vim /etc/fstab
Add the lines to the configuration (see example above) to automount your drives
Save and close the fstab file
Test your configuration by executing a mount all command

mount -a

If your drive correctly mounted to the mount point you specified (/media/1TB) and it is mounted as read-write, you're golden.

If you are using multiple drives, then simply add additional lines to the fstab to cover all of the drives you want to automount. Now our storage will automatically mount read-write at startup.

Part 1.3 - Samba

In my build, I wanted to share out any drives mounted in /media/ in case I added drives in the future. Additionally, accessibility rather than security was the priority for me. Because of that I set up Samba to allow anonymous access to any device on my LAN. If you're cool with that approach, follow along. If not, you may have to do some research on how to set up Samba to your liking.

Install samba

sudo apt-get install samba
Open the samba config file in your favorite text editor

sudo vim /etc/samba/smb.conf

Find the line that says “security = user” and update the file with the following lines

# "security = user" is always a good idea. This will require a Unix account
# in this server for every user accessing the server. See
# /usr/share/doc/samba-doc/htmldocs/Samba3-HOWTO/ServerType.html
# in the samba-doc package for details.
   security = share
   guest account = nobody

Scroll to the bottom of the Samba configuration file and add a new section for your new share. See my configuration for example

[shared]
     comment = Shared 1TB drive on Raspberry Pi
     path = /media/1TB/
     browseable = yes
     read only = no
     guest ok = yes

Test your configuration by executing testparm and examining the output

pi@raspberrypi ~ $ testparm
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[homes]"
Processing section "[printers]"
Processing section "[print$]"
Processing section "[shared]"
WARNING: The security=share option is deprecated
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions

Assuming your test went well, reload samba

sudo /etc/init.d/samba reload
On another device, try to mount the share. On Windows, you can just open the start menu and type “\yourservername” or “\your.server.IP.address” On a Mac, open Finder, go to the “Go” dropdown menu at the top, and choose connect to server and enter “smb://yourservername” or “smb://your.server.IP.address” If prompted for a user account, choose Guest.

Courtesy of Mobilefish.com

OK, this is already over 1000 words long so let's take a breather and continue with XBMC in Part 2.

Part 2 - XBMC

Part 2.1 - Installing XBMC

Installing XBMC is usually pretty easy, but since we're using Raspbian we can't use the normal packages. Fortunately for me and anyone who is trying to accomplish this, Michael Gorven has already figured this out for us and even hosts the packages we need. Go ahead and open a new window or tab and browse to http://michael.gorven.za.net/raspberrypi/xbmc . The author has already done a great job of documenting this process so I won't duplicate it here. Once your installation is finished, test that you can start XBMC by executing 'sudo xbmc-standalone' and seeing if XBMC appears on your display.

This is perfect, except that XBMC is now tied to our session. Now if you're directly connected to the Pi with a keyboard, this is irrelevant. If, like me, you're doing all of your Pi work via SSH, XBMC will die once you terminate your SSH session. Let's set up XBMC as a service so we can start/stop it gracefully and allow it to run after our SSH session ends. Additionally, if you're having issues with XBMC not starting at boot, setting up a service can ensure that XBMC is started for the right run levels.

Part 2.2 - Setting up XBMC as a Service

The XBMC Wiki has a good article for how to set up XBMC as a service, but I want to expand on it a little bit.

Create the init script file
```
vim /etc/init.d/xbmc
```

Paste the following code into your vim session:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          xbmc
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       starts instance of XBMC using start-stop-daemon and xinit
# Short-Description: starts instance of XBMC
### END INIT INFO

############### EDIT ME ##################

# path to xinit exec
DAEMON=/usr/bin/xinit

# startup args
DAEMON_OPTS=" /usr/bin/xbmc --standalone -- :0"

# script name
NAME=xbmc

# app name
DESC=XBMC

# user
RUN_AS=pi

# Path of the PID file
PID_FILE=/var/run/xbmc.pid

############### END EDIT ME ##################

test -x $DAEMON || exit 0

set -e

case "$1" in
  start)
        echo "Starting $DESC"
        start-stop-daemon --start -c $RUN_AS --background --pidfile $PID_FILE  --make-pidfile --exec $DAEMON -- $DAEMON_OPTS
        ;;
  stop)
        echo "Stopping $DESC"
        start-stop-daemon --stop --pidfile $PID_FILE
        ;;

  status)
        echo "Checking status of $DESC..."
        PID=$(cat $PID_FILE)
        # This line is for debugging purposes only
        # echo "PID is $PID"
        if ps -p $PID > /dev/null ; then
                echo "XBMC is running with PID $PID"
        else
                echo "XBMC doesn't seem to be running"
        fi
        ;;

  restart|force-reload)
        echo "Restarting $DESC"
        start-stop-daemon --stop --pidfile $PID_FILE
        sleep 5
        start-stop-daemon --start -c $RUN_AS --background --pidfile $PID_FILE  --make-pidfile --exec $DAEMON -- $DAEMON_OPTS
        ;;
  *)
        N=/etc/init.d/$NAME
        echo "Usage: $N {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

Save and quit out of vim. There are a few differences between my init script and the one provided on the XBMC wiki. For one, my “runas” user is set to “pi” - you’ll want to set this to whatever user you installed XBMC with to avoid permissions issues. Additionally, my init script will take “status” as an argument to the init script and will return information about whether or not XBMC is running.

Change the script’s permissions to make it executable
```
chmod a+x /etc/init.d/xbmc
```

Test that the init script is working:

pi@raspberrypi ~ $ sudo /etc/init.d/xbmc start
Starting XBMC
pi@raspberrypi ~ $ /etc/init.d/xbmc status
Checking status of XBMC...
XBMC is running with PID 10986
pi@raspberrypi ~ $ /etc/init.d/xbmc stop
Stopping XBMC
pi@raspberrypi ~ $ /etc/init.d/xbmc status
Checking status of XBMC...
XBMC doesn't seem to be running

If you want XBMC to start when the OS boots, update the rc.d scripts. Unless you have a specific use case, using the “defaults” argument will work for you.
```
sudo update-rc.d xbmc defaults
```
If you want to be thorough, restart your Raspberry Pi and verify that your disk has been mounted (via fstab from Part 1) and that XBMC has started.
You’re done. Time to enjoy.

Part 2.3 - XBMC Remote

If you're going to manage your XBMC/NAS/Pi remotely, I would also suggest downloading the XBMC Remote App for your Adroid or iPhone. To enable remote control on XBMC, go to System -> Settings -> Services -> Remote Control and enable both "Allow programs on this system to control XBMC" and "Allow programs on other systems to control XBMC."

Courtesy of xbmc.org

Assuming you're on the same wireless network, you should be able to connect to the XBMC with your remote app using your Pi's IP address and the default username and password of 'xbmc.'

Well at nearly 2000 words, this has been a whopper of a post. Hopefully this write up has helped you one way or another, and let me know if you have any questions. Enjoy your new combination NAS and HTPC.

Powering a Raspberry Pi

2013-12-18T00:23:08+00:00

I recently started building out a combination NAS and HTPC on a Raspberry Pi and came across some interesting information about powering Raspberry Pi's. While this isn't a project post per se, I thought it was interesting enough to share and wanted to send this out as sort of a PSA for the users out there who don't know the potential danger of "backpowering" a Raspberry Pi.

For those who don't already have one, here are a few suggestions for where to get a Raspberry Pi.

Where to get a Raspberry Pi

If you have some of the peripherals (SD card, HDMI cable, micro USB power supply), you can get started with a standalone Model B Raspberry Pi for as low as $35.

Element14 - $35
AlliedElec - $35
SparkFun - $39.95
Amazon (Prime eligible) - $40.60
DealExtreme (free shipping) - $55.70

If you would rather get a starter kit, the prices are pretty reasonable and usually includes an SD card pre-loaded with the Raspberry Pi New Out Of Box Software (NOOBS). Here are two good starter kits.

Raspberry Pi Model B, Case, Micro USB Power Supply, SD Card with NOOBS, and HDMI Cable via Amazon (Prime eligible) - $61.95
Raspberry Pi Model B, Case, Micro USB Power Supply, SD Card with NOOBS, and HDMI Cable via Canakit - $69.95

Power Consumption

As an Arduino die hard, I didn't think I really needed a Raspberry Pi for anything. I was sold once I realized I could run all of my home web server functions on the 3.5 watt ARM processor instead of 85 watt Intel chip in my desktop. The Raspberry Pi will actually pay for itself in about 4 months.

Utility cost of running home server on full x86 desktop (Intel dual core, ATI Radeon HD GPU, 3 HDDs) [high estimate]

(Power draw x hours per week x weeks per year x cost per kilowatt hour) / (1000 watts per kW x 0.85 efficiency x 12 months a year) = estimated cost per month

(150 watts x 168 hours x 52 weeks x $0.10 per kW hour) / 1000 watts per kW x 0.85 efficiency x 12 months a year) = $12.85 per month

Utility cost of running home server on Raspberry Pi and one external hard drive (5 watts for Raspberry Pi, 15 watts for HDD under load) [high estimate]

(20 watts x 168 hours x 52 weeks x $0.10 per kW hour) / 1000 watts per kW x 0.85 efficiency x 12 months a year) = $1.71 per month

Approximate cost savings: $11.14 per month

Powering the Pi and the Dangers of "Backpowering"

The Pi only has two USB ports, so obviously many folks incorporate a powered USB hub to expand its capabilities. At some point, someone discovered that the powered hubs were actually supplying power to the Pi via the uplink between the hub and the Pi. Now you can power your peripherals and the Pi from one outlet. Cool right?

It's pretty cool until you accidentally fry your Pi or start a fire. The power supply connection on the Pi includes a polyfuse to protect against overvoltages (see diagram below).

The USB data connections do not have an inline polyfuse! If you want to run your Pi off of a powered USB hub, you'll need to do two things.

Acquire a powered USB hub that specifically does NOT supply 'backpower' via the datalink/uplink/data port. If you want to play it safe, check out the list of compatible powered USB hubs that have been tested by the community. I chose to use the Plugable 7-port USB hub.
Run a USB-to-micro-USB cable from one of the outputs on your USB hub back to the power input connection on the Pi. This will properly route your +5v power through the polyfuse, protecting your Pi.

See picture below for a visual explanation.

Obviously you still need to plug your powered USB hub into a power source. Now your Raspberry Pi will run off of the powered USB hub, allowing you to safely run your Pi and your peripherals from one power supply.

Anyway, just wanted to share a couple of lessons learned while getting started and highlight the details behind "backpowering."

How do you power your Pi?

Thematic Password Generator

2013-11-02T17:20:32+00:00

I've been wanting to build a thematic password generator for some time. Basically, choose some sort of "theme" and a few password strength parameters, and it will generate random passwords for you. Will come back to update this post and expand on themes shortly.

Pick a theme:

Words that will be used for this password (editable):

Minimum password length:

Embed numbers?

Capitalize first letters?

Use exact length?

Password will appear here...

Here’s the code if you’re interested:

<p>Pick a theme:</p>
<select id="themeSelector" onchange="populateDictionaryTextArea()">
  <option disabled="disabled" selected="selected">Choose a theme...</option>
  <option value="flowers">Flowers</option>
  <option value="starwars">Star Wars</option>
  <option value="seaAnimals">Sea Animals</option>
  <option value="jellyBellyFlavors">Jelly Belly Flavors</option>
  <option value="countries">Countries</option>
  <option value="phonetic">Phonetic (military) Alphabet</option>
</select>
<p>Words that will be used for this password (editable):</p>
<p><textarea id="dictionaryTextArea" cols="50" rows="5"></textarea></p>
<p>Minimum password length:</p>
<input id="passwordLength" type="text" /><br />

<div>
<p class="align-left" style="padding-right:10px;">Embed numbers?</p>
<p class="checkboxThree align-left">
  <input type="checkbox" id="embedNumbersChkBox" />
  <label for="embedNumbersChkBox"></label>
</p>
</div>

<div style="clear:both;"></div>

<div>
<p class="align-left" style="padding-right:10px;">Capitalize first letters?</p>
<p class="checkboxThree align-left">
  <input type="checkbox" id="capitalizeChkBox" />
  <label for="capitalizeChkBox"></label>
</p>
</div>

<div style="clear:both;"></div>

<div>
<p class="align-left" style="padding-right:10px;">Use exact length?</p>
<p class="checkboxThree align-left">
  <input type="checkbox" id="exactLengthChkBox" />
  <label for="exactLengthChkBox"></label>
</p>
</div>

<div style="clear:both;"></div>

<p><input type="button" onclick="makePasswd()" class="btn btn--large btn--success" value="Generate" /></p>

<p><textarea id="output" cols="50" disabled="disabled" rows="5">Password will appear here...</textarea></p>

<script>
function makePasswd() {
    var password = '';
    var desiredLength = document.getElementById('passwordLength').value;
    var embedNumbers = document.getElementById('embedNumbersChkBox').checked;
    var capitalize = document.getElementById('capitalizeChkBox').checked;
    var exactLength = document.getElementById('exactLengthChkBox').checked;

  var words = document.getElementById('dictionaryTextArea').value.split(" ");
    while (password.length < desiredLength) {
        var word = words[Math.floor(Math.random()*words.length)];
        if(capitalize){
            word = word.charAt(0).toUpperCase() + word.slice(1);
        }
        password += word;
        if(embedNumbers && (password.length < desiredLength)){
            password += Math.floor(Math.random()*51);
        }
    }
    if(exactLength){
        password = password.slice(0,desiredLength);
    }
    var actualLength = password.length;
    password += "\n\nPasswword is " + actualLength + " characters long.";

  document.getElementById('output').value = password;
}

function populateDictionaryTextArea(){
  var dictionaries = { samuelLJackson : ["bullshit","shoot","bitch","kneecaps","goddamn","money","motherfucker","Winston","motherfucker","shot","understand","asses","dead","fucking","fried","chicken","shit","transitional","kill","shit","case","dumbass","duty","booty","motherfucker","righteous","ezekiel","brother","tyranny","filthy","pumpkin-pie","AK-47","kill","absolutely","snakes","plane","zeus","fuck","shove","lightning-bolt","ass","goddamn","shark","enough"], pokemon: ["Bulbasaur","Ivysaur","Venusaur","Charmander","Charmeleon","Charizard","Squirtle","Wartortle","Blastoise","Caterpie","Metapod","Butterfree","Weedle","Kakuna","Beedrill","Pidgey","Pidgeotto","Pidgeot","Rattata","Raticate","Spearow","Fearow","Ekans","Arbok","Pikachu","Raichu","Sandshrew","Sandslash","Nidoran","Nidorina","Nidoqueen","Nidoran","Nidorino","Nidoking","Clefairy","Clefable","Vulpix","Ninetales","Jigglypuff","Wigglytuff","Zubat","Golbat","Oddish","Gloom","Vileplume","Paras","Parasect","Venonat","Venomoth","Diglett","Dugtrio","Meowth","Persian","Psyduck","Golduck","Mankey","Primeape","Growlithe","Arcanine","Poliwag","Poliwhirl","Poliwrath","Abra","Kadabra","Alakazam","Machop","Machoke","Machamp","Bellsprout","Weepinbell","Victreebel","Tentacool","Tentacruel","Geodude","Graveler","Golem","Ponyta","Rapidash","Slowpoke","Slowbro","Magnemite","Magneton","Farfetchd","Doduo","Dodrio","Seel","Dewgong","Grimer","Muk","Shellder","Cloyster","Gastly","Haunter","Gengar","Onix","Drowzee","Hypno","Krabby","Kingler","Voltorb","Electrode","Exeggcute","Exeggutor","Cubone","Marowak","Hitmonlee","Hitmonchan","Lickitung","Koffing","Weezing","Rhyhorn","Rhydon","Chansey","Tangela","Kangaskhan","Horsea","Seadra","Goldeen","Seaking","Staryu","Starmie","Mr. Mime","Scyther","Jynx","Electabuzz","Magmar","Pinsir","Tauros","Magikarp","Gyarados","Lapras","Ditto","Eevee","Vaporeon","Jolteon","Flareon","Porygon","Omanyte","Omastar","Kabuto","Kabutops","Aerodactyl","Snorlax","Articuno","Zapdos","Moltres","Dratini","Dragonair","Dragonite","Mewtwo","Mew†","Chikorita","Bayleef","Meganium","Cyndaquil","Quilava","Typhlosion","Totodile","Croconaw","Feraligatr","Sentret","Furret","Hoothoot","Noctowl","Ledyba","Ledian","Spinarak","Ariados","Crobat","Chinchou","Lanturn","Pichu","Cleffa","Igglybuff","Togepi","Togetic","Natu","Xatu","Mareep","Flaaffy","Ampharos","Bellossom","Marill","Azumarill","Sudowoodo","Politoed","Hoppip","Skiploom","Jumpluff","Aipom","Sunkern","Sunflora","Yanma","Wooper","Quagsire","Espeon","Umbreon","Murkrow","Slowking","Misdreavus","Unown","Wobbuffet","Girafarig","Pineco","Forretress","Dunsparce","Gligar","Steelix","Snubbull","Granbull","Qwilfish","Scizor","Shuckle","Heracross","Sneasel","Teddiursa","Ursaring","Slugma","Magcargo","Swinub","Piloswine","Corsola","Remoraid","Octillery","Delibird","Mantine","Skarmory","Houndour","Houndoom","Kingdra","Phanpy","Donphan","Porygon2","Stantler","Smeargle","Tyrogue","Hitmontop","Smoochum","Elekid","Magby","Miltank","Blissey","Raikou","Entei","Suicune","Larvitar","Pupitar","Tyranitar","Lugia†","Ho-Oh†","Celebi†","Treecko","Grovyle","Sceptile","Torchic","Combusken","Blaziken","Mudkip","Marshtomp","Swampert","Poochyena","Mightyena","Zigzagoon","Linoone","Wurmple","Silcoon","Beautifly","Cascoon","Dustox","Lotad","Lombre","Ludicolo","Seedot","Nuzleaf","Shiftry","Taillow","Swellow","Wingull","Pelipper","Ralts","Kirlia","Gardevoir","Surskit","Masquerain","Shroomish","Breloom","Slakoth","Vigoroth","Slaking","Nincada","Ninjask","Shedinja","Whismur","Loudred","Exploud","Makuhita","Hariyama","Azurill","Nosepass","Skitty","Delcatty","Sableye","Mawile","Aron","Lairon","Aggron","Meditite","Medicham","Electrike","Manectric","Plusle","Minun","Volbeat","Illumise","Roselia","Gulpin","Swalot","Carvanha","Sharpedo","Wailmer","Wailord","Numel","Camerupt","Torkoal","Spoink","Grumpig","Spinda","Trapinch","Vibrava","Flygon","Cacnea","Cacturne","Swablu","Altaria","Zangoose","Seviper","Lunatone","Solrock","Barboach","Whiscash","Corphish","Crawdaunt","Baltoy","Claydol","Lileep","Cradily","Anorith","Armaldo","Feebas","Milotic","Castform","Kecleon","Shuppet","Banette","Duskull","Dusclops","Tropius","Chimecho","Absol","Wynaut","Snorunt","Glalie","Spheal","Sealeo","Walrein","Clamperl","Huntail","Gorebyss","Relicanth","Luvdisc","Bagon","Shelgon","Salamence","Beldum","Metang","Metagross","Regirock","Regice","Registeel","Latias","Latios","Kyogre","Groudon","Rayquaza","Jirachi†","Deoxys†","Turtwig","Grotle","Torterra","Chimchar","Monferno","Infernape","Piplup","Prinplup","Empoleon","Starly","Staravia","Staraptor","Bidoof","Bibarel","Kricketot","Kricketune","Shinx","Luxio","Luxray","Budew","Roserade","Cranidos","Rampardos","Shieldon","Bastiodon","Burmy","Wormadam","Mothim","Combee","Vespiquen","Pachirisu","Buizel","Floatzel","Cherubi","Cherrim","Shellos","Gastrodon","Ambipom","Drifloon","Drifblim","Buneary","Lopunny","Mismagius","Honchkrow","Glameow","Purugly","Chingling","Stunky","Skuntank","Bronzor","Bronzong","Bonsly","Mime Jr.","Happiny","Chatot","Spiritomb","Gible","Gabite","Garchomp","Munchlax","Riolu","Lucario","Hippopotas","Hippowdon","Skorupi","Drapion","Croagunk","Toxicroak","Carnivine","Finneon","Lumineon","Mantyke","Snover","Abomasnow","Weavile","Magnezone","Lickilicky","Rhyperior","Tangrowth","Electivire","Magmortar","Togekiss","Yanmega","Leafeon","Glaceon","Gliscor","Mamoswine","Porygon-Z","Gallade","Probopass","Dusknoir","Froslass","Rotom","Uxie","Mesprit","Azelf","Dialga","Palkia","Heatran","Regigigas","Giratina","Cresselia","Swoobat"], flowers: ["alstroemeria","amaranthus","amaryllis","anemone","anthurium","asters","buplerum","callalilies","callalily","carnation","mum","chrysanthemum","coxcomb","daisy","curly-willow","daffodils","dahlias","delphinium","gerbera","ginger","gladiouli","heather","heliconia","hyacinth","hydrangeas","hypericum","iris","kangaroo-paw","larkspur","leptospermum","liatris","lilies","limonium","lisianthus","asters","narcissus","orchids","pearl-blossom","peonies","poinsettia","protea","quince","ranunculus","roses","snapdragons","soldaster","sunflowers","tulips","viburnum","waxflower"], starwars: ["tan-tan","luke","han","skywalker","solo","bacta","at-at","at-st","wookie","wampa","hoth","empire","dark","side","force","leia","princess","speeder","tie-fighter","deathstar","feeling","lightsaber","x-wing","b-wing","y-wing","a-wing","millenium","falcon","destroyer","star","imperial","sith","jedi","knight","tie-bomber","blaster","shield","shields","endor","coruscant","clones","deathsticks","kaashyyk","bothans","proton","torpedo","attack","formation","spacestation","chewbacca","c3po","r2d2","droid","anakin","obi-wan","kenobi","maul","darth","vader","palpatine","emperor","operational","alderaan"], seaAnimals: ["anemone","eel","carp","pike","walleye","clownfish","kraken","barracuda","shark","angelfish","squid","octopus","crab","lobster","urchin","whale","dolphin","seasnake","manatee","cuttlefish","conch","cod","salmon","tuna","barnacle","anchovy","albacore","coral","seal","flounder","goldfish","grouper","guppy","haddock","halibut","herring","jellyfish","lamprey","manta-ray","stingray","minnow","mollusk","mussel","narwhal","nautilus","nettlefish","orca","otter","oyster","plankton","piranha","sardine","shrimp","starfish","sturgeon","trout","turtle","walrus"], jellyBellyFlavors: ["cream","soda","root","beer","berry","blue","blueberry","bubble","gum","buttered","popcorn","cantaloupe","cappuccino","caramel","corn","chili","mango","chocolate","pudding","cinnamon","coconut","cotton","candy","crushed","pineapple","dr.pepper","french","vanilla","green","apple","island","punch","juicy","pear","kiwi","lemon","drop","lemon","lime","licorice","mango","margarita","mixed","berry","smoothie","orange","sherbet","peach","piña","colada","plum","pomegranate","raspberry","red","apple","sizzling","cinnamon","sour","cherry","strawberry","cheesecake","strawberry","daiquiri","strawberry","jam","sunkist","lemon","sunkist","lime","sunkist","orange","sunkist","pink","grapefruit","sunkist","tangerine","toasted","marshmallow","top","banana","tutti-fruitti","very","cherry","watermelon","wild","blackberry"], countries:["afghanistan","albania","algeria","american-samoa","andorra","angola","anguilla","antigua","barbuda","argentina","armenia","aruba","australia","austria","azerbaijan","bahamas","bahrain","bangladesh","barbados","belarus","belgium","belize","benin","bermuda","bhutan","bolivia","bosnia-herzegovina","botswana","bouvet-island","brazil","brunei","bulgaria","burkina-faso","burundi","cambodia","cameroon","canada","cape-verde","cayman-islands","central-african-republic","chad","chile","china","christmas-island","cocos-(keeling)-islands","colombia","comoros","zaire","cook-islands","costa-rica","croatia","cuba","cyprus","czech-republic","denmark","djibouti","dominica","dominican-republic","ecuador","egypt","el-salvador","equatorial-guinea","eritrea","estonia","ethiopia","falkland-islands","faroe-islands","fiji","finland","france","french-guiana","gabon","gambia","georgia","germany","ghana","gibraltar","greece","greenland","grenada","guadeloupe","guam","guatemala","guinea","guinea-bissau","guyana","haiti","holy-see","honduras","hong-kong","hungary","iceland","india","indonesia","iran","iraq","ireland","israel","italy","ivory-coast","jamaica","japan","jordan","kazakhstan","kenya","kiribati","kuwait","kyrgyzstan","laos","latvia","lebanon","lesotho","liberia","libya","liechtenstein","lithuania","luxembourg","macau","macedonia","madagascar","malawi","malaysia","maldives","mali","malta","marshall-islands","martinique","mauritania","mauritius","mayotte","mexico","micronesia","moldova","monaco","mongolia","montenegro","montserrat","morocco","mozambique","myanmar","namibia","nauru","nepal","netherlands","antilles","new-caledonia","new-zealand","nicaragua","niger","nigeria","niue","norfolk-island","north-korea","northern-mariana-islands","norway","oman","pakistan","palau","panama","papua-new-guinea","paraguay","peru","philippines","pitcairn-island","poland","polynesia","portugal","puerto-rico","qatar","reunion","romania","russia","rwanda","saint-helena","saint-kitts-and-nevis","saint-lucia","saint-pierre-and-miquelon","saint-vincent-and-grenadines","samoa","san-marino","sao-tome-and-principe","saudi-arabia","senegal","serbia","seychelles","sierra-leone","singapore","slovakia","slovenia","solomon-islands","somalia","south-africa","south-georgia-and-south-sandwich-islands","south-korea","south-sudan","spain","sri-lanka","sudan","suriname","svalbard-and-jan-mayen-islands","swaziland","sweden","switzerland","syria","taiwan","tajikistan","tanzania","thailand","timor-leste","togo","tokelau","tonga","trinidad-and-tobago","tunisia","turkey","turkmenistan","turks-and-caicos-islands","tuvalu","uganda","ukraine","united-arab-emirates","united-kingdom","united-states","uruguay","uzbekistan","vanuatu","venezuela","vietnam","virgin-islands","wallis-and-futuna-islands","yemen","zambia","zimbabwe"], phonetic : ["alpha","bravo","charlie","delta","echo","foxtrot","gulf","hotel","india","juliet","kilo","lima","mike","november","oscar","papa","quebec","romeo","sierra","tango","uniform","victor","whiskey","x-ray","yankee","zulu"] };

  var dictionarySelect   = document.getElementById('themeSelector');
  var dictionaryName    = dictionarySelect.options[dictionarySelect.selectedIndex].value;

  //alert(dictionaryName);

  var dictionaryTextArea  = document.getElementById('dictionaryTextArea');

  var dictionary = dictionaries[dictionaryName];

  dictionaryTextArea.value = "";

  for(var i = 0; i < dictionary.length; i++){
    dictionaryTextArea.value += dictionary[i];
    dictionaryTextArea.value += " ";
  }
}
</script>

Wedding Web Site Released

2013-09-29T01:17:28+00:00

For those of you out there planning a wedding, you may have thought about setting up a website. Something to answer those common questions guests may have so that you can focus on all of the other aspects of wedding planning. There are some pretty slick WYSIWYG sites out there like Wedding Wire, The Knot, and Wedding Channel if you want to use a pre-built theme and just punch in your relevant information.

My wife and I couldn't find a theme and a site that fit our needs perfectly so I ended up building one for us. Check it out or see the screen shot below.

The site is built with Bootstrap 2 to provide the responsive framework and base CSS, Galleria for the photo gallery. I used PHP so I could re-use the header code on the other pages. If you're looking at the site on your computer (as opposed to phone/tablet), go to the accommodations or schedule page and try re-sizing your browser to make it smaller - you'll see the two-column layout collapse into a mobile-friendly single column layout.

Anyway, this is less of a tutorial post and more of a "use my code if it will help you jump start your project."

Demo GitHub Repo

Enjoy and let me know if you have any questions.

Querying vCloud Director API with Ruby Script

2013-08-01T01:28:09+00:00

Well it's been awhile since I've posted since I started working on HireUpJob, but this one's good. I recently had a use case where I needed to get the virtual hardware version of a client machine provisioned through vCloud Director. Now, that information is not accessible anywhere on the client machine, so I had to look to the vCloud Director API for it. The process for querying the API also had to be automated for future provisions, so it had to be scripted. Enough back story and context, lets look at the ruby code.

#Need to import a few ruby packages
require 'net/http'
require 'uri'
require 'openssl'
#Here we are ignoring any issues that may arise with SSL certificates
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

#main fetch function takes an initial URL to fetch from and a limit for number of redirects to follow
def fetch(uri_str, limit = 10)
  raise ArgumentError, 'HTTP redirect too deep' if limit == 0

  #Parse URL into URI object
  uri = URI.parse(uri_str)
  #Create new Net::HTTP object
  http = Net::HTTP.new(uri.host, uri.port)
  #Set the http object to use SSL
  http.use_ssl = true
  #Ignore issues with SSL certs
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  #Start the http session
  http.start()
  #Create an HTTP POST request to /api/sessions to authenticate
  req = Net::HTTP::Post.new('/api/sessions')
  #Set HTTP basic auth variables of username and password
  req.basic_auth '<your_username>@<your_org>', '<your_password>'
  req.add_field 'Accept:', 'application/*+xml;version=1.5'
  #Send the request we created earlier, and capture the HTTP response
  response = http.request(req)
  #Start case to handle various HTTP responses
  case response
  #If authentication is successful, a token is returned in the response header. So if we get an HTTP 200 response from the vCloud server, then we can grab the "x-vcloud-authorization" token from the header
  when Net::HTTPSuccess     then
  authKey = response.header['x-vcloud-authorization']
  #if we get an HTTPfound instead, we are probably getting redirected and should try to follow it. You may need to tweak this section to suit your use case
  when Net::HTTPFound    then
        #call the main fetch function again and decrement the overall redirect limit
  fetch(response['location'], limit - 1)
  #If we get an HTTP redirection, we are definitely getting redirected and should follow it
  when Net::HTTPRedirection then fetch(response['location'], limit - 1)
  else
    #something went wrong - spit out whatever the response error is
    puts response.error!
  end

  #Set the headers as a variable to be re-used later, including the authorization token we grabbed earlier
  headers = {
    'Accept' => 'application/*+xml;version=1.5',
  'x-vcloud-authorization' => "#{authKey}"
  }

  #Build a new request, this time an HTTP GET, to grab a value from the API. Supply both the API call and the headers variable we just put together
  req = Net::HTTP::Get.new('/api/vApp/vm-insert-your-alphanumeric-UUID-here/virtualHardwareSection', headers)
  #send the new request and grab the response again
  response = http.request(req)
  case response
  when Net::HTTPSuccess     then
        #In my case, the value I wanted was returned in the XML body that was returned, so if the server responds with HTTP 200 I just grab the entire response body and pass it to a variable
  result = response.body()
  #same as previous
  when Net::HTTPFound    then
  fetch(response['location'], limit - 1)
  #same as previous
  when Net::HTTPRedirection then fetch(response['location'], limit - 1)
  else
    puts response.error!
  end

  #Finally, build one more request to send an HTTP DELETE call to /api/sessions to kill your session
  req = Net::HTTP::Delete.new('/api/sessions', headers)
  http.request(req)

  #Again, in my case, I wanted to confirm that the VM was using a specific virtual hardware version (vmx-08), so I did a quick string match to confirm this
  result.include?('<vssd:VirtualSystemType>vmx-08</vssd:VirtualSystemType>')

end

print fetch('https://<your_vcloud_server>:443')

If you're using my code more or less verbatim and everything executes properly, you should get a simple "true" or "false" as output. Now, this script is for a fairly specific use case, but I think the real potential value here is to use this enable Opscode Chef (written in Ruby) to communicate with vCloud Director. I'll be sure to post again as I develop some use cases.

In case you have a very different use case but need some help, here are some of the pages I referred to for writing this script:

Exploring the vCloud REST API - Part I and Part II
vCloud Director REST API Reference
Ruby - Net::HTTP - following redirects

Hope this helps some of you out there and let me know if you have any questions.

HTML5 Canvas and a Horace Dediu-esque Chart

2013-05-13T13:00:32+00:00

A few months back, I came across this chart illustrating Apple's segment revenues on Reddit:

This is a great chart because it has impact with just a glance, and yet yields more solid data the longer you examine it. Ever since I saw this chart I wanted to build my own for my department, for startups, and offer it up to anyone else who might want to build a similar chart.

Horace Dediu of Asymco.com put this together using a chart-building iPad app called Perspective. Now, I should have just downloaded the damn app and made some similar charts and been happy. Instead I built my own from scratch using HTML5's canvas tag. Check out the demo.

Demo GitHub Repo

In case you didn't notice it right away (I didn't either), Horace Dediu's chart uses the same colors to represent both a product's revenues as well as their cost of goods sold, giving you an at-a-glance idea of what a product's margin is. I replicated this feature in my tool as well.

Instead of a long-winded walk through of how I built the tool, I'm just going to share the source here and add some extra commenting for clarity.

<html>
  <head>
  <title>Finance Chart</title>
    <style>
      body {
        margin: 20px;
        padding: 20px;
      }
    </style>
  </head>
  <body>
  <!-- A basic HTML form for getting custom inputs from users. All of these values will be sent as HTTP GET variables in the URL-->
  <!-- From there, the JavaScript will fetch these variables out of the URL-->
  <form method="GET" action="html5canvas.html">
    <label>Item 1 Label</label><input type="text" name="item1Label" /><br/>
    <label>Item 1 Revenues</label><input type="number" name="item1Rev" /><br/>
    <label>Item 1 COGS</label><input type="number" name="item1Cos" /><br/>
    <label>Item 2 Label</label><input type="text" name="item2Label" /><br/>
    <label>Item 2 Revenues</label><input type="number" name="item2Rev" /><br/>
    <label>Item 2 COGS</label><input type="number" name="item2Cos" /><br/>
    <label>Item 3 Label</label><input type="text" name="item3Label" /><br/>
    <label>Item 3 Revenues</label><input type="number" name="item3Rev" /><br/>
    <label>Item 3 COGS</label><input type="number" name="item3Cos" /><br/>
    <label>Operating Expense</label><input type="number" name="opEx" /><br/>
    <label>Scale ($M, $k, etc)</label><input type="text" name="scaleLabel" /><br/>
    <input type="Submit" value="Submit" /><br/>
  </form>
  <!-- The canvas tag - the only important thing here is to set your width and height as necessary-->
    <canvas id="myCanvas" width="1000" height="1200"></canvas>
  <script>
  // A handy little function that I picked up at http://papermashup.com/read-url-get-variables-withjavascript/
  // The function simply parses a GET variable out of the URL using REGEX and returns it
  function getUrlVars() {
    var vars = {};
    var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi, function(m,key,value) {
      vars[key] = value;
    });
    return vars;
  }
  </script>
    <script>
    // Grabbing all of the user form variables out of the URL. For integer values, we need to cast them as an int datatype
    // because by default, as part of the URL string, they exist as strings
    var item1Label  =(getUrlVars()["item1Label"]);
    var item1Rev  =parseInt((getUrlVars()["item1Rev"]));
    var item1Cos  =parseInt((getUrlVars()["item1Cos"]));

    var item2Label  =(getUrlVars()["item2Label"]);
    var item2Rev  =parseInt((getUrlVars()["item2Rev"]));
    var item2Cos  =parseInt((getUrlVars()["item2Cos"]));

    var item3Label  =(getUrlVars()["item3Label"]);
    var item3Rev  =parseInt((getUrlVars()["item3Rev"]));
    var item3Cos  =parseInt((getUrlVars()["item3Cos"]));

    var opExInput  =parseInt((getUrlVars()["opEx"]));

    var scaleLabelInput = (getUrlVars()["scaleLabel"]);

    // We calculate the total revenue right away. Total revenue should be the largest number in any practical use case
    // and we will use it to set the scale for all of the rest of the variables
    var totalRev   = item1Rev + item2Rev + item3Rev;

    // Calculate and set variables for scaling purposes
    var scaler = (Math.ceil(100/totalRev));
    var totalRevScaled = (scaler*totalRev);

    // Scale label that will be used later - a conversion on the size of the scale bar will happen at the time of drawing the scale
    var scaleValue = '= 10 ' + scaleLabelInput;

    // Scale all of the item revenue and cost of goods solds values, and set them to new variables (unnecessary artifact of old code)
    var revenueItem1  = scaler*item1Rev;
    var revenueItem2  = scaler*item2Rev;
    var revenueItem3  = scaler*item3Rev;
    // Enter all revenues into an array to iterate through later. Do the same for labels
    var revenues = new Array(revenueItem1,revenueItem2,revenueItem3);
    var revenuesLabels = new Array(item1Label,item2Label,item3Label);

    //more detailed level COGS variables
    var cosItem1  = scaler*item1Cos;
    var cosItem2  = scaler*item2Cos;
    var cosItem3  = scaler*item3Cos;
    var cosItems  = new Array(cosItem1,cosItem2,cosItem3);
    // This array is redundant with revenuesLabels, another artifact
    var cosItemsLabels = new Array(item1Label,item2Label,item3Label);

    // Set total COS, scale the Operating Expense value, and set operating income (aka net income in this case)
    // to total revenue minus all other costs and operating expense. We use scaled values in this case because
    // we don't actually care about the dollar value, just that the cart is proportionally correct
    var cos        = cosItem1+cosItem2+cosItem3;
    var opEx      = scaler*opExInput;
    var operatingIncome  = totalRevScaled - (cosItem1+cosItem2+cosItem3) - opEx;
    // Set an array to iterate through for expense categories
    var expenseCats = new Array(operatingIncome,cos,opEx);
    var expenseCatsLabels = new Array('Operating Income','COGS','Operating Expense');

    // generic canvas variable setting
    var canvas = document.getElementById('myCanvas');
    var context = canvas.getContext('2d');

    // Generic variables for storing location information - these will be used, incremented, and re-used to act as a "ceiling"
    // for the bars in the chart
    var globalVerticalAdjustment  = 30;
    var revenueVerticalAdjustment  = 50;

    // Generic color array for assigning different colors to all blocks
    var colors = new Array('black','orange','red','yellow','green','blue','purple','gray','white');
    var c = 0;

    // Begin "painting" revenue bars
    // Grab the number of items in the revenues array
    var length = revenues.length;

    //for debugging only
    //alert("revenues array has "+length+" items");

    for (var i = 0; i < length; i++) {
      //debugging code
      //for debugging only
      //alert("global vert adj"+globalVerticalAdjustment);
      //draw rectangle for revenue source
      context.beginPath();
      // Set the upper left top corner. We will always use 230 as the horizontal value so all of the bars align
      // globalVerticalAdjustment will be incremented each time we paint a bar. 20 represents the width of the bar
      // in pixels, and the last argument is the height of the bar
      context.rect(230, globalVerticalAdjustment, 20, (revenues[i]));
      // Set the color of the bar, and increment the color array for the next color selection
      context.fillStyle = colors[c];
      c++;
      context.fill();
      // Set line color and width, then paint
      context.lineWidth = 2;
      context.strokeStyle = 'black';
      context.stroke();

      // Set label for revenue sources
      context.textAlign = 'right';
      context.fillStyle = 'black';
      context.font = "14pt sans-serif";
      // Here we use the revenuesLabel array to get the correct item name, and then set it to a location to correspond
      // with the location of the bar
      context.fillText(revenuesLabels[i], 250, globalVerticalAdjustment-10);

      // If this is NOT the first iteration, then we need to account for the previous bars heights for spacing reasons
      if(i!=0){
        revenueVerticalAdjustment = (revenueVerticalAdjustment + revenues[i-1]);
        //for debugging only
        //alert("rev vertical adj = "+revenueVerticalAdjustment);
      }

      // Drawing the top lines that connect with the total revenue bar
      context.beginPath();
      context.moveTo(250, (globalVerticalAdjustment));
      // The curve is simply a line based on 4 points - the origin, which is set in the 'moveTo' function, two points that dictate
      // which direction to curve the line (they act almost like gravity wells, if that helps your understanding), and an ending
      // point for the line
      context.bezierCurveTo(260, revenueVerticalAdjustment, 380, revenueVerticalAdjustment, 420, revenueVerticalAdjustment);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      // Drawing bottom lines to connect with total revenue bar - same functions, just different arguments
      context.beginPath();
      context.moveTo(250, (globalVerticalAdjustment+revenues[i]));
      context.bezierCurveTo(260, revenueVerticalAdjustment+revenues[i], 380, revenueVerticalAdjustment+revenues[i], 420, revenueVerticalAdjustment+revenues[i]);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      globalVerticalAdjustment += (revenues[i] + 50);
    }

    //Draw vetical revenue bar here - here we don't need an iterative function, because there always be one and only one revenue line
    context.beginPath();
    context.rect(420, 50, 20, (totalRevScaled));
    context.fillStyle = colors[c];
    c++;
    context.fill();
    context.lineWidth = 2;
    context.strokeStyle = 'black';
    context.stroke();

    // Create the text label for the total revenue bar
    context.fillStyle = 'black';
    context.font = "14pt sans-serif";
    context.textAlign = 'left';
    context.fillText("Total Revenue", 380, 40);

    // Resetting generic variables for location informatino
    globalVerticalAdjustment    = 30;
    expenseCatVerticalAdjustment  = 50;

    length = expenseCats.length;

    //for debugging only
    //alert("expense category array has "+length+" items");

    for (var i = 0; i < length; i++) {
      //for debugging only
      //alert("global vert adj "+globalVerticalAdjustment);

      // Same set of functions as the revenues section - drawing rectangles in the same way, just with different values
      context.beginPath();
      context.rect(620, globalVerticalAdjustment, 20, (expenseCats[i]));
      context.fillStyle = colors[c];
      c++;
      context.fill();
      context.lineWidth = 2;
      context.strokeStyle = 'black';
      context.stroke();

      // Set label for expense category
      context.fillStyle = 'black';
      context.font = "14pt sans-serif";
      context.fillText(expenseCatsLabels[i], 620, globalVerticalAdjustment-10);

      if(i!=0){
        expenseCatVerticalAdjustment = (expenseCatVerticalAdjustment + expenseCats[i-1]);
        //for debugging only
        //alert("rev vertical adj = "+expenseCatVerticalAdjustment);
      }

      // Drawing top lines to connect with total revenue bar
      // Only difference is here is we are drawing the lines from right to left now, instead of left to right. This was done to keep the code
      // as consistent as possible
      context.beginPath();
      context.moveTo(620, (globalVerticalAdjustment));
      context.bezierCurveTo(600, expenseCatVerticalAdjustment, 440, expenseCatVerticalAdjustment, 440, expenseCatVerticalAdjustment);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      //drawing bottom lines to connect with total revenue bar
      context.beginPath();
      context.moveTo(620, (globalVerticalAdjustment+expenseCats[i]));
      context.bezierCurveTo(600, expenseCatVerticalAdjustment+expenseCats[i], 440, expenseCatVerticalAdjustment+expenseCats[i], 440, expenseCatVerticalAdjustment+expenseCats[i]);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      globalVerticalAdjustment += (expenseCats[i] + 50);

    }

    //resetting generic variables for location informatino
    globalVerticalAdjustment    = 30 + operatingIncome;
    expenseCatVerticalAdjustment  = 50 + operatingIncome + 30;
    c = 0;

    length = cosItems.length;

    //for debugging only
    //alert("expense category array has "+length+" items");

    for (var i = 0; i < length; i++) {
      //for debugging only
      //alert("global vert adj "+globalVerticalAdjustment);

      // Again, drawing rectangles in the exact same way as before, just different values
      context.beginPath();
      context.rect(820, globalVerticalAdjustment, 20, (cosItems[i]));
      context.fillStyle = colors[c];
      c++;
      context.fill();
      context.lineWidth = 2;
      context.strokeStyle = 'black';
      context.stroke();

      //set label for revenue source
      context.fillStyle = 'black';
      context.font = "14pt sans-serif";
      context.fillText(cosItemsLabels[i], 820, globalVerticalAdjustment-10);

      if(i!=0){
        expenseCatVerticalAdjustment = (expenseCatVerticalAdjustment + cosItems[i-1]);
        //for debugging only
        //alert("rev vertical adj = "+expenseCatVerticalAdjustment);
      }

      // Drawing top lines to connect with total revenue bar
      // As before with the expense categories, we draw the lines from right to left, this time aligning with the COS bar instead of
      // the total revenue bar. Also, to account for the variable height and location of the COS bar, we now use the expenseCatVerticalAdjustment
      // variable as the ceiling for the drawn lines.
      context.beginPath();
      context.moveTo(820, (globalVerticalAdjustment));
      context.bezierCurveTo(800, expenseCatVerticalAdjustment, 640, expenseCatVerticalAdjustment, 640, expenseCatVerticalAdjustment);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      // Drawing bottom lines to connect with total revenue bar
      // Same as above, except we now add the scaled value of cosItems[i] to expenseCatVerticalAdjustment to draw the bottom line
      context.beginPath();
      context.moveTo(820, (globalVerticalAdjustment+cosItems[i]));
      context.bezierCurveTo(800, expenseCatVerticalAdjustment+cosItems[i], 640, expenseCatVerticalAdjustment+cosItems[i], 640, expenseCatVerticalAdjustment+cosItems[i]);
      context.lineWidth = 1;
      context.strokeStyle = 'black';
      context.stroke();

      globalVerticalAdjustment += (cosItems[i] + 50);

    }

    // Begin scale section
    context.beginPath();
    // Here we draw a rectangle that is a scaled 10 units tall (relative to user inputs). The top of the scale rectangle will always be
    // 150 pixels below the total revenue bar
    context.rect(420, totalRevScaled+150, 20, 10*scaler);
    context.fillStyle = colors[c];
    c++;
    context.fill();
    context.lineWidth = 2;
    context.strokeStyle = 'black';
    context.stroke();

    //set label for scale
    context.fillStyle = 'black';
    context.font = "14pt sans-serif";
    // Set the scale label to line up with the scale rectangle
    context.fillText(scaleValue, 450, totalRevScaled+160);

    </script>
  </body>
</html>

It could still use some polish and pizzazz, but overall I'm pretty pleased with the tool. I've also learned with some time, math skills, and trial and error, you can draw just about anything in HTML5's canvas tag. If you are looking to build something using canvas, I'd highly recommend checking out HTML5 Canvas Tutorials. With almost zero knowledge of the canvas tag, I was able to knock out this project in just a few hours with the help of their tutorials.

As always, hope you enjoyed the post and let me know if you have any questions!

Web Scraping, PHP, and Wheel of Fortune

2013-04-17T02:13:22+00:00

Be advised This post is quite old (17 Apr 2013) and any code may be out of date. Proceed with caution.

My fiancee loves Wheel of Fortune and watches whenever she can. The clever folks over at Sony (producers of Wheel of Fortune) introduced a loyalty program called "Wheel Watchers." People who sign up get a "Spin ID," and if your Spin ID is chosen for a given episode, you win one of the prizes they gave away on the show. Only catch is you have to watch every night.

This is a lot of random background information, but there's a reason. My fiancee asked me to write an application that would check the Spin ID every day and notify us if we won. This seemed like a great reason to learn some web scraping with PHP (although you could probably do this in just about any language).

Pulling Raw Data

First things first, you need a decent website to scrape the information from! Strangely, the official Wheel of Fortune site doesn't offer up the winning spin IDs. Luckily, there are a handful of websites that do report the winning numbers. For my application, I'll be using http://wheeloffortuneclub.blogspot.com

We'll pull the entire web page first and then parse for the Spin ID later. In PHP, this is quite simple:

<?php
$data = file_get_contents('http://wheeloffortuneclub.blogspot.com/');
?>

Keep in mind, many site admins will not take kindly to you scraping their site especially if you're doing it frequently. In this example, we'll only need to scrape the information once a day, so it shouldn't be a problem.

Parsing for the Spin ID

When scraping web pages, regular expressions come in handy. To pull the specific data you're looking for, you may need to use a clever combination of identifying content as well as identifying HTML tags and attributes to retrieve the data. In the case of the Spin ID, it's two capital letters and 6-7 numbers. This is a pretty specific format, so it'll be pretty easy to pull using regex.

Now, regex syntax can be tough if you don't use it on a regular basis. Regex 101 is an awesome site to use as a reference for regex syntax or to test your expressions.

For the Spin ID, our regular expression is "[A-Z][A-Z]\d{6,}". This translates to two capital letters ([A-Z][A-Z]) followed by 6 or more numbers (\d{6,}). We'll create a variable for our regular expression and parse our previously fetched web page to look for the expression using preg_match, which will return the first match:

$regex = '/[A-Z][A-Z]\d{6,}/';
preg_match($regex,$data,$match);

We're passing three parameters to preg_match - the pattern we're seeking, the string subject, and an output variable ($match in this case).

The $match variable is actually an array, and so we'll refer to the first object in the array to get the string. For now, we'll just echo the variable out to the page to confirm that everything's working!

$regex = '/[A-Z][A-Z]\d{6,}/';
preg_match($regex,$data,$match);
echo $match[0];

So the complete code looks like this:

<?php
$data = file_get_contents('http://wheeloffortuneclub.blogspot.com/');
$regex = '/[A-Z][A-Z]\d{6,}/';
preg_match($regex,$data,$match);
echo $match[0];
?>

Automation with Cron and Email

So we've got a PHP page that will parse and return the most recent winning Spin ID. So what? I could have just browsed to the Spin ID website and gotten the same information. We need to automate the parsing and compare it to our specific Spin ID (to see if we're a winner) and contact us if it's a match. If it's a match, we'll send an email to notify us. Here's the code in it's entirety:

<?php
$data = file_get_contents('http://wheeloffortuneclub.blogspot.com/');
$regex = '/[A-Z][A-Z]\d{6,}/';
preg_match($regex,$data,$match);
echo $match[0];
echo "<br />";
if ($match[0]=="KW6426861"){
  echo "Match!!!";
  mail("alexdglover@gmail.com","Spin ID Match!!!","Spin ID Match!!!");
}
else{
  echo "Not a match!!!";
}
?>

The Spin ID "KW6426861" was the most recent winning ID at the time I wrote this script, and so the check resolved to true and sent me a convenient notification email. Awesome. Now, to finish our project, we just need to regularly execute the PHP script with a Cron job. If you're using your home server to host, you can just write a crontab entry using php -f /path/to/your/php/script.php and execute it at whatever interval you want.

If you are using hosting externally, most CPanels will offer a cron functionality. Again, you just need to provide the command ("php -f" in this case), the path to your php script, and then your interval. I used " 0 */12 * * * " to check every 12 hours.

That's it! A very simple but powerful PHP script in just 14 lines of code!

Feb 1, 2015 Update: For those who don't have their own servers or can't be bothered to build their own SpinID monitoring service, check out WheelNotify.com - for just $1/month, the service notify you via email, text, and/or phone if your SpinID is ever a winner on Wheel Of Fortune. Awesome!

To Do List App

2013-04-03T22:01:55+00:00

Be advised This post is quite old (03 Apr 2013) and any code may be out of date. Proceed with caution.

Update Feb 2017 I’ve happily used this TODO app for the past ~4 years, but I’m taking it down with the rest of my old PHP hosting. In conjunction, I’ve removed the demo-related content from this post. For anyone interested, I’m now using Wunderlist

Short post, as this isn't really a tutorial. I was recently checking out ToDoMVC to look at some JavaScript frameworks, and I really liked the To Do list app...

I've been looking for something similar for awhile. Looked at Evernote and other apps, but nothing was as simple and lightweight as this app. Only problem was that the items stored in the app didn't persist, as there was not database/storage behind the app. So I went ahead and wrote my own.

GitHub Repo

In my haste to modify the to do list app to save values, I skipped the part where I should have learned one of the new JavaScript frameworks highlighted on ToDoMVC. I actually used the template provided in the ToDoMVC package and used the asynchronous database call setup I outlined in a related post.

The bad news - this isn't a secure application, SQL injection attacks will succeed here. Also, some of the icons don't seem to work outside of Chrome.

The good news - this application supports multiple clients via the use of a "poor man's pull." Basically, if User A and User B are writing to dos in the app on two separate computers, they will see each other's notes in near real time. This is possible by using JavaScript to call the DB query every X number of milliseconds. Here's a code snippet if you're having a hard time understanding:

<script>
setInterval( function(){ queryDB() }, 5000);   //Call the queryDB() function every 5 seconds
</script>

<script type="text/javascript">        
function queryDB()
{
  $.ajax({
    type: "POST",
    url: "dbEngine.php",
    data: "keyword=read",
    success: function(server_response)
    {
      $('#todo-list').html(server_response).show();
    }
  })
}
</script>

The app also works on mobile phones, which is great.

If you want to set up your own instance of this app, you can download the code from GitHub. Obviously you'll need your own hosting and a MySQL database, but the rest should be pretty easy to figure out.

This isn't a fully finished product by any means, but I wanted to put it out there in case there's someone like me who needs a simple to-do list app.

Asynchronous Database Operations with PHP and jQuery

2013-03-26T13:00:57+00:00

Be advised This post is quite old (26 Mar 2013) and any code may be out of date. Proceed with caution.

In a recent work project, we had to do some simple database calls to update records. I was going to do a simple form with a submit button, but decided checked in with one of my coworkers (who's a far better developer than I) to see what he thought. Since we were developing a 'flashier' application, he pushed me to do it all asynchronously with jQuery. I had a hard time finding some good examples at the time, so I decided to create some demos and do a write up.

Before we start looking at the code, I need to mention two things. For this walk-through, we'll start with a simple PHP form and add in jQuery components. If this doesn't interest you, feel free to skip down to the final section. Second, if you want to follow my example to the letter, you'll need to setup a database (I'm using MySQL) and add a table 'customer' with the following structure:

Here's a quick script to create the table if you're feeling lazy:

CREATE TABLE `customer` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `Last_Name` varchar(50) COLLATE latin1_general_ci NOT NULL,
  `First_Name` varchar(50) COLLATE latin1_general_ci NOT NULL,
  `Phone_Number` varchar(10) COLLATE latin1_general_ci NOT NULL,
  `Street_Address` varchar(50) COLLATE latin1_general_ci NOT NULL,
  `City` varchar(50) COLLATE latin1_general_ci NOT NULL,
  `State` varchar(25) COLLATE latin1_general_ci NOT NULL,
  `ZIP` varchar(5) COLLATE latin1_general_ci NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci AUTO_INCREMENT=19 ;

Of course you can use a different database, different table name, columns etc., just keep that in mind as you write your code.

Part 1 - Simple PHP Form and Submit Button

Before we add anything new, let's just get a basic page working. This page will have a form with a text field input for each field in the database (except the ID, since it auto-increments), and a submit button. The page will also have a simple table to show the records currently in the database. If you're totally new to PHP forms and MySQL, check out the W3Schools PHP tutorials on forms, $_GET/$_POST, INSERT and SELECT.

<head>

<?php
$con=mysqli_connect("<hostname>","<username>","<password>","<database_name>");
// Check connection
if (mysqli_connect_errno())
  {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
  }

if (isset($_POST["lastName"])) {
  $sql="INSERT INTO customer (Last_Name, First_Name, Phone_Number, Street_Address, City, State, Zip) VALUES ('$_POST[lastName]', '$_POST[firstName]', '$_POST[phoneNumber]', '$_POST[address]', '$_POST[city]', '$_POST[state]', '$_POST[zip]')";
  if (!mysqli_query($con,$sql))
  {
    die('Error: ' . mysqli_error());
  }
  echo "1 record added";
}
?>

<title>Alex Glover's Portfolio Site</title>
</head>
<body>
<h1>Asynchronous Database Calls Demo</h1>

<form action="asynchDatabaseDemo.php" method="post">
Last Name: <input type="text" name="lastName"><br />
First Name: <input type="text" name="firstName"><br />
Phone: <input type="text" name="phoneNumber"><br />
Address: <input type="text" name="address"><br />
City: <input type="text" name="city"><br />
State: <input type="text" name="state"><br />
ZIP: <input type="text" name="zip"><br />
<button type="submit">Submit</button>
</form>

<br />
<br />
<br />

<table>
<tr><th>Last Name</th><th>First Name</th><th>Phone Number</th><th>Address</th><th>City</th><th>State</th><th>Zip</th></tr>

<?php
/* Get the entire appgen table to fill out the table on the page */
$query = "SELECT * FROM customer order by Last_Name desc;";
if($result = $con->query($query)) {
  while ($row = $result->fetch_row()){
    echo "<tr>";
    echo "<td>" . $row[1] . "</td>";
    echo "<td>" . $row[2] . "</td>";
    echo "<td>" . $row[3] . "</td>";
    echo "<td>" . $row[4] . "</td>";
    echo "<td>" . $row[5] . "</td>";
    echo "<td>" . $row[6] . "</td>";
    echo "<td>" . $row[7] . "</td>";
    echo "</tr>";
  }
  $result->close();
}
?>
</table>
</body>
</html>

So now we can insert new records, and see all the records in the table. This is all Web Development 101, but it's important that we've built a working foundation. This way if we have issues later we can narrow it down to JavaScript or database issues.

Part 2 - Introducing jQuery and Serialize

OK, we're going to take a small step. Instead of using a button within a form to submit the page and all of the form values, we'll use JavaScript. Conveniently, jQuery has a serialize function we can use to grab all of the input values at once and format them in a convenient string. Then we'll use an ajax call to submit that pre-formatted string to the server.

I haven't removed any of the original code, simply added one JavaScript function insert() and one anchor tag that will call the insert function. The insert() function will actually POST the values back to the server, back to the very same page (see the url: "asynchDatabaseDemo2.php" line). Since our PHP is looking for POST values, it will 'catch' them and insert them into the database. Let's review the code:

<head>

<?php
$con=mysqli_connect("<hostname>","<username>","<password>","<database_name>");
// Check connection
if (mysqli_connect_errno())
  {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
  }

if (isset($_POST["lastName"])) {
  $sql="INSERT INTO customer (Last_Name, First_Name, Phone_Number, Street_Address, City, State, Zip) VALUES ('$_POST[lastName]', '$_POST[firstName]', '$_POST[phoneNumber]', '$_POST[address]', '$_POST[city]', '$_POST[state]', '$_POST[zip]')";
  if (!mysqli_query($con,$sql))
  {
    die('Error: ' . mysqli_error());
  }
  echo "1 record added";
  //mysqli_close($con);
}
?>

<title>Alex Glover's Portfolio Site</title>
<script src="http://code.jquery.com/jquery-1.9.1.js"></script>
<script type="text/javascript">    
function insert()
{
  var formValues = $('form').serialize();

  alert(formValues);
  $.ajax({
    type: "POST",
    url: "asynchDatabaseDemo2.php",
    data: formValues
  })
}
</script>

</head>
<body>
<h1>Asynchronous Database Calls Demo</h1>

Search:<input type="text" onkeydown="search()"><button type="submit"></button>

<br />
<br />
<br />

<form action="asynchDatabaseDemo.php" method="post">
Last Name: <input type="text" name="lastName"><br />
First Name: <input type="text" name="firstName"><br />
Phone: <input type="text" name="phoneNumber"><br />
Address: <input type="text" name="address"><br />
City: <input type="text" name="city"><br />
State: <input type="text" name="state"><br />
ZIP: <input type="text" name="zip"><br />
<button type="submit">Submit</button>
<a href="#" onclick="insert()">jQuery Insert</button>
</form>

<br />
<br />
<br />

<table>
<tr><th>Last Name</th><th>First Name</th><th>Phone Number</th><th>Address</th><th>City</th><th>State</th><th>Zip</th></tr>

<?php
/* Get the entire appgen table to fill out the table on the page */
$query = "SELECT * FROM customer order by Last_Name desc;";
if($result = $con->query($query)) {
  while ($row = $result->fetch_row()){
    echo "<tr>";
    echo "<td>" . $row[1] . "</td>";
    echo "<td>" . $row[2] . "</td>";
    echo "<td>" . $row[3] . "</td>";
    echo "<td>" . $row[4] . "</td>";
    echo "<td>" . $row[5] . "</td>";
    echo "<td>" . $row[6] . "</td>";
    echo "<td>" . $row[7] . "</td>";
    echo "</tr>";
  }
  $result->close();
}
?>
</table>
</body>
</html>

Simple right? The data is inserted, we see the alert that shows the serialized string... but the table doesn't show the new values. No worries, we just need to make the table operate asynchronously as well.

Part 3 - Asynchronous Search Results

This is a slightly more complicated step, so a couple quick notes before we get started:

JavaScript can't call the PHP directly to re-query the database; JavaScript is a client-side language, PHP is a server-side language. However, we can 'cheat' by using HTTP POST or GET requests to act as a bridge between the two. To do this we need to create a separate PHP page to handle the searching, called "asynchDatabaseDemo_DBEngine.php". This page will echo table tags identical to our original table structure
We'll add a new JavaScript function, searchDB(), that will POST to the PHP page dedicated to querying the database. Then we'll add a search field that calls the searchDB() function. Finally, we'll modify the insert() function to call the searchDB() function - that way the table will be refreshed as soon as the insert completes
The ajax call within the searchDB() function has an additional attribute, 'success' that will handle the server response by placing the HTML within the table tags (by referencing the ID '#searchResults')

Alright, hopefully that will make the code make a little more sense. Let's look at the asynchDatabaseDemo_DBEngine.php file to start:

<?php
$con=mysqli_connect("<hostname>","<username>","<password>","<database_name>");
// Check connection
if (mysqli_connect_errno())
{
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
}

if (isset($_POST["keyword"]))
{
  $keyword = $_POST["keyword"];
  $query = "SELECT * FROM customer where Last_Name like '%$keyword%' order by Last_Name desc;";
  if($result = $con->query($query))
  {
    echo "<tr><th>Last Name</th><th>First Name</th><th>Phone Number</th><th>Address</th><th>City</th><th>State</th><th>Zip</th></tr>";

    while ($row = $result->fetch_row()){
      echo "<tr>";
      echo "<td>" . $row[1] . "</td>";
      echo "<td>" . $row[2] . "</td>";
      echo "<td>" . $row[3] . "</td>";
      echo "<td>" . $row[4] . "</td>";
      echo "<td>" . $row[5] . "</td>";
      echo "<td>" . $row[6] . "</td>";
      echo "<td>" . $row[7] . "</td>";
      echo "</tr>";
    }
    $result->close();
  }
  else
  {
    echo 'No Results for :"'.$_POST['keyword'].'"';
  }

}
?>

And finally the aynchDatabaseDemo_frontEnd.php code:

<head>

<?php
$con=mysqli_connect("<hostname>","<username>","<password>","<database_name>");
// Check connection
if (mysqli_connect_errno())
  {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
  }

if (isset($_POST["lastName"])) {
  $sql="INSERT INTO customer (Last_Name, First_Name, Phone_Number, Street_Address, City, State, Zip) VALUES ('$_POST[lastName]', '$_POST[firstName]', '$_POST[phoneNumber]', '$_POST[address]', '$_POST[city]', '$_POST[state]', '$_POST[zip]')";
  if (!mysqli_query($con,$sql))
  {
    die('Error: ' . mysqli_error());
  }
  echo "1 record added";
  //mysqli_close($con);
}
?>

<title>Alex Glover's Portfolio Site</title>
<script src="http://code.jquery.com/jquery-1.9.1.js"></script>
<script type="text/javascript">    
function insert()
{
  var formValues = $('form').serialize();

  alert(formValues);

  $.ajax({
    type: "POST",
    url: "asynchDatabaseDemo_frontEnd.php",
    data: formValues
  })

  queryDB();

  $(this).closest('form').find("input[type=text], textarea").val("");
}
</script>
<script type="text/javascript">    
function queryDB()
{
  var keyword = $("#searchField").val();

  $.ajax({
    type: "POST",
    url: "asynchDatabaseDemo_DBEngine.php",
    data: "keyword=" + keyword,
    success: function(server_response)
    {
      $('#searchResults').html(server_response).show();
    }
  })
}
</script>
</head>
<body>
<h1>Asynchronous Database Calls Demo</h1>

Search:<input type="text" onkeydown="queryDB()" id="searchField"><a href="#" onclick="queryDB()">Search</a>

<br />
<br />
<br />

<form action="asynchDatabaseDemo.php" method="post">
Last Name: <input type="text" name="lastName"><br />
First Name: <input type="text" name="firstName"><br />
Phone: <input type="text" name="phoneNumber"><br />
Address: <input type="text" name="address"><br />
City: <input type="text" name="city"><br />
State: <input type="text" name="state"><br />
ZIP: <input type="text" name="zip"><br />
<button type="submit">Submit</button>
<a href="#" onclick="insert()">jQuery Insert</button>
</form>

<br />
<br />
<br />

<table id="searchResults">
<tr><th>Last Name</th><th>First Name</th><th>Phone Number</th><th>Address</th><th>City</th><th>State</th><th>Zip</th></tr>

<?php
/* Get the entire appgen table to fill out the table on the page */
$query = "SELECT * FROM customer order by Last_Name desc;";
if($result = $con->query($query)) {
  while ($row = $result->fetch_row()){
    echo "<tr>";
    echo "<td>" . $row[1] . "</td>";
    echo "<td>" . $row[2] . "</td>";
    echo "<td>" . $row[3] . "</td>";
    echo "<td>" . $row[4] . "</td>";
    echo "<td>" . $row[5] . "</td>";
    echo "<td>" . $row[6] . "</td>";
    echo "<td>" . $row[7] . "</td>";
    echo "</tr>";
  }
  $result->close();
}
?>
</table>
</body>
</html>

Done! This tutorial came out a little more scatter-brained than I had hoped, so feel free to comment/contact me if you have any questions. As always, I hope this helps.

New Site Launch + 6 WordPress.com Migration Tips

2013-03-10T03:50:09+00:00

Like all good things, my time on WordPress.com's hosting has come to an end. Don't get me wrong - anyone toying with the idea for a blog should strongly consider it, as it's well-managed and does 90% of what any user needs it to do. It's a powerful tool for the uninitiated...

So as one of the 'initiated,' I had to go to my own hosting to take full control. For subscribers of my old site, you've been migrated (this is probably obvious since this post has been emailed to you) with the rest of the site. If you're just visiting the site, consider registering - it's free, hides all advertisements, and signs you up to receive future posts by email.

Now, in the process of migrating to my own WordPress instance, I've learned a number of valuable things. And what's a better way to transition to a new site than to cover the lessons learned from the migration? I present to you, the 6 WordPress.com Migration Tips!

6 WordPress.com Migration Tips

Now some of these tips are more generic, some are specific to migrating FROM WordPress.com hosting. I'm not going to cover the hosting setup or WordPress install, but leave a comment if you have questions related to those processes.

Assuming you want to keep all of your pages/posts/images, your first step should be to do an export/import. On your WordPress.com site's dashboard, go to Tools --> Export

WordPress.com will present you with 2 options, a free XML export or a "Guided Transfer" for a whopping $129... just do the XML export. Remember where you saved your XML export! Next, go to your new WordPress instance dashboard and choose Tools --> Import. Click on the WordPress option, and install the necessary plugin. Then upload your XML file. The import should include all pages, posts, and even images/media.

Now, that's all pretty useless, "WordPress-101" information. Here's the important tip - if you do the import, and you seem to be missing pages/posts, try 2 things:
- Re-run the import - the WordPress import function is smart enough to skip any posts/pages that already exist based on permalink.
- Go to your "All Posts" view and look for "Deleted" posts. A good number of my posts were marked as deleted for some reason. Then just select all of the posts that should NOT be deleted, and click restore!
Like the stats, WP Reader, and "Freshly Pressed" features that WordPress.com offers? Not to fear - install JetPack! JetPack allows you to connect your stand-alone instance of WordPress back to the WordPress.com's admin panel, allowing you to continue using all of the functionality you've come to know and love. I was pretty psyched that I was still able to use the "Views by Country" feature after going to my own WP instance.
Set up a subscription service. JetPack offers a form for readers to subscribe to your blog like when it was hosted at WordPress.com. Alternatively, if you're looking for some additional control and features (or if you don't want to rely on WordPress.com), check out Subscribe2. It's a no muss, no fuss plugin that allows users to subscribe to your blog with their email address. Unlike the JetPack service, you can control which posts do and do not get pushed out to your subscribers. Any user that is registered with your instance will also receive post notifications, which can come in handy if you're using a membership plugin.
If you're changing themes, try to decide on one before you spend time customizing widgets! Oftentimes when you activate a new theme, your widgets will get deactivated. If you do change themes after customizing, don't re-create all of the widgets - they should be in the Inactive Widget area at the bottom of the widgets screen! I made the mistake of re-creating everything. The widget settings should still be saved, you just need to drag them into the sidebar to reactivate them.
If you've left WordPress.com so that you can start advertising on your blog, I strongly suggest you check out Easy AdSense. It allows you to easily inject Google ads into posts, sidebars, and headers/footers. Easy AdSense also attempts to limit the number of ads on any given page to comply with Google's policies. Best of all, it's free!
This is a very broad statement, but start thinking more like a sysadmin - you're no longer operating in the safe sandbox of WordPress.com. When you're on your own hosting, you're fully capable of deleting your entire instance and all of your content (or otherwise royally screwing things up). Start taking backups, make sure your domain name doesn't expire or get sold to another party, and keep your instance and plug-ins updated.

I hope some of these tips help you with your migration, and I hope you enjoy the new site.

Arduino Project 5: Internet Control without Ethernet Shield

2013-02-18T15:00:30+00:00

I'm getting ready to start an Arduino home automation project, so I started looking at ways to interface with an Arduino across the internet. That way I'll be able to control all of the lights, locks, etc. anywhere I have internet access.

Now, the obvious answer was to buy an Ethernet shield, but I already run a home web server so that seemed unnecessary. I saw a few solutions using Processing or Python scripts, but that seemed unnecessarily complicated. It took a fair amount of digging and brainstorming, but I've found an ultra-easy, ultra-flexible, and ultra-fast solution. Ultra.

There are a few major assumptions here.

You have a computer running a web server that is accessible from the open internet, using a static IP address or a Dynamic DNS name. Your computer is either directly connected to your ISP and using a public IP address or your router is set up for port forwarding. If you don't have this set up yet, just Google it - there are a lot of tutorials that explain how to set up a home web server.
Your home web server has PHP. If not, this will still work, you'll just have to re-write it in Java or whatever server-side language you're using.
The Arduino will be plugged into the web server via USB.

Here's the basic concept: the Arduino can't read files from the server via the USB serial connection, so the server will have to "push" the message. The server side code (PHP, Java, whatever you choose) cannot talk directly to the serial COM port, so we need a local script on the web server that can talk to the serial (USB) port. Last, the Arduino sketch has to be written so it can "catch" and process the message.

In this example, we'll just create a barebones web page for controls that will turn an LED on and off via the Arduino. Let's start with the sketch. The sketch is just listening on the serial connection for a 1 or 0. If it receives a 1 it will turn the LED on, if it receives a 0 it will turn the LED off.

/*
Alex Glover
February 2013
*/

void setup() {
  Serial.begin(9600);
  //set the LED pin to OUTPUT
  pinMode(13, OUTPUT);
}

void loop() {
  //wait until the serial connection is open
  while (Serial.available() ==0);

  //read from the serial connection; the - '0' is to cast the values as the int and not the ASCII code
  int val = Serial.read() - '0';

  //print to the console for testing
  Serial.println(val);

  //if we've recieved a '1', turn on the LED and print a message
  if(val==1){
    Serial.println("Received a 1");
    digitalWrite(13, HIGH);
  }
  //if we've recieved a '0', turn off the LED and print a message
  if(val==0){
    Serial.println("Received a 0");
    digitalWrite(13, LOW);
  }
}

Pretty straightforward, right? OK, now we need two scripts, one to send a '1' and one to send a '0'. In Windows, simply create a text file (call it whatever you want), and give it a .bat extension. In my setup, my files are called serial_out_0.bat and serial_out_1.bat. Each script has only one line of code.

ECHO 0 > COM3:

and

ECHO 1 > COM3:

Note that you might have to change the COM designation. You can check which COM your Arduino is connected to by looking in the Arduino IDE under Tools --> Serial Port. If you're not using Windows, you should be able to do this pretty easily in a shell script. At this point you can test to see if the batch scripts will turn the LED off and on. Also ensure that the web server will be able to execute these scripts (don't assign user-specific privileges or save them in protected directories). If you're unsure, just put these scripts in the same web root directory where you'll host your web page.

Easy right? Alright, the last piece is the web form. All you need are buttons within a form that will then execute the batch scripts we wrote earlier. Easiest solution is to use 'submit' buttons and then check for which post variables are set. The rest of the code is very straightforward so I'll let it speak for itself.

<head>
<?php
if(isset($_POST['submitOn'])) {
  exec(‘C:\xampp\htdocs\Home_Automation\serial_out_1.bat’);
}
else if(isset($_POST['submitOff'])) {
  exec(‘C:\xampp\htdocs\Home_Automation\serial_out_0.bat’);
}
?>
</head>
<body>
  <form action=”control.php” method=”post”>
    <input type=”submit” name=”submitOn” value=”Submit On”>
    <input type=”submit” name=”submitOff” value=”Submit Off”>
  </form>
</body>

</html>

You'll have to change the paths to correspond with the location of your scripts, but otherwise that's it. Add some CSS if you want to add some polish to your controls. This is a very simple example, but you should be able to adapt this code to any project.

Arduino Project 4: Tilt To Unlock

2013-02-07T23:52:52+00:00

This is my favorite Arduino project to date. After some basic experimenting with an accelerometer, I had an idea. What if you used the accelerometer to look for a combination of movements? Instead of using a combination lock or a traditional key lock, you could just tilt the object in a set pattern to unlock it. Here's what I ended up with:

Pretty sweet if you ask me.

Note - in spite of my terrible, coffee-fueled, shakey-handed videography, a lot of this post will be in video form.

Now this post is based on my Acclerometer Primer, so if you're following along and want to build one of these for yourself you should go back and build the primer.

Parts Used (beyond what was used in the primer)

Amico DC 12V Open Frame Type Solenoid for Electric Door Lock
SainSmart 4-Channel 5V Relay Module (any 5v relay will work, we won't use all 4 channels)
12v Power Supply (you can use a battery or wall adapter)
Some sort of box/container - something larger than 4in x 4in x 4in, otherwise you may have trouble getting all of the components in

Phase 1 - Adapting the Primer To Recognize a Combination

Before we try to add any new hardware, let's modify the Accelerometer Primer code to look for a pattern of movements. We've already got the mechanism for recognizing simple single movements, so all we need is to add some code to track multiple movements relative to each other. I just added a new integer variable called "currentMove." currentMove will get incremented each time a correct move in the sequence is made. To thwart people from just shaking the box or trying random patterns, each incorrect move will reset currentMove to 1. Let's see it in action:

Here's my code:

/*####################################################################
Tilt To Unlock
Alex Glover
February 2013
Combination to unlock box is Y+ (left), X+ (forward), Y+ (left), X- (back), Y+ (forward)
Based on autoCalibration.ino, provided by virtuabotix
#######################################################################*/
#include "Accelerometer.h"</code>
Accelerometer myAccelerometer = Accelerometer();
int yPosPin = 11;
int yNegPin = 13;
int xPosPin = 10;
int xNegPin = 9;
int zPin = 12;
int currentMove = 1;
int resetTrigger = 0;
void setup()
{
  Serial.begin(9600);
  //Connect up the following pins and your power rail
  // SL GS 0G X Y Z
  myAccelerometer.begin(3, 4, 5, A0, A1, A2);
  pinMode(yPosPin, OUTPUT);
  pinMode(yNegPin, OUTPUT);
  pinMode(xPosPin, OUTPUT);
  pinMode(xNegPin, OUTPUT);
  pinMode(zPin, OUTPUT);
  //calibration performed below
  Serial.println("Please place the Accelerometer on a flatnLevel surface");
  delay(2000);//Give user 2 seconds to comply
  myAccelerometer.calibrate();
}
void loop()
{
  delay(20);//delay for readability
  resetTrigger++; //increment the reset trigger
  myAccelerometer.read();
  if(myAccelerometer._Zgs <= -0.9){ digitalWrite(zPin, HIGH); } else if(myAccelerometer._Zgs > -0.9){
    digitalWrite(zPin, LOW);
  }
  if(myAccelerometer._Xgs >= 0.5){
    digitalWrite(xPosPin, HIGH);
    if(currentMove==2){
      Serial.println("Second move successful");
      currentMove++;
      delay(1000);
    }
    else if(currentMove==5){
      Serial.println("Fifth move successful - opening");
      currentMove++;
      delay(1000);
    }
    else{
      Serial.println("WRONG MOVE!");
      currentMove=1;
    }
  }
  else if(myAccelerometer._Xgs < 0.5){
    digitalWrite(xPosPin, LOW);
  }
  if(myAccelerometer._Xgs <= -0.5){
    digitalWrite(xNegPin, HIGH);
    if(currentMove==4){
      Serial.println("Fourth move successful");
      currentMove++;
      delay(1000);
    }
    else{
      Serial.println("WRONG MOVE!");
      currentMove=1;
    }
  }
  else if(myAccelerometer._Xgs > -0.5){
    digitalWrite(xNegPin, LOW);
  }
  if(myAccelerometer._Ygs >= 0.5){
    digitalWrite(yPosPin, HIGH);
    if(currentMove==1){
      Serial.println("First move successful");
      currentMove++;
      delay(1000);
    }
    else if(currentMove==3){
      Serial.println("Third move successful");
      currentMove++;
      delay(1000);
    }
    else{
      Serial.println("WRONG MOVE!");
      currentMove=1;
    }
  }
  else if(myAccelerometer._Ygs < 0.5){
    digitalWrite(yPosPin, LOW);
  }
  if(myAccelerometer._Ygs <= -0.5){
    digitalWrite(yNegPin, HIGH);
    Serial.println("WRONG MOVE!");
    currentMove=1;
  } else if(myAccelerometer._Ygs > -0.5){
    digitalWrite(yNegPin, LOW);
  }
  if(resetTrigger==3000){ //do this every minute (3000 iterations * 0.02 seconds = 60 seconds)
    currentMove=1; //reset currentMove back to 1, so user can try to do the sequence again
    resetTrigger=0; //set the resetTrigger back to 0 so it can start another iteration to 3000
  }
}

Perfect! Now we can recognize a combination of movements. Now all we need to do is something meaningful when the combination is recognized.

Phase 2 - Incorporating The Cigar Box, Relay, and 12v Door Latch

Time for the hardware. Now, at this point we should be pretty confident in our code (we are confident in our code, right?) so we can get rid of the LEDs and all of the associated jumper wires. Next, switch your Virtuabotix Accelerometer from using the 5v power from the Arduino to using the 3.3v power from the Arduino (you'll need to change the connection at both ends, since the accelerometer has connections for both). We need to switch to 3.3v because our relay is going to use the 5v power from the Arduino. That being said, connect the 5v power to the VCC pin on the relay and also connect the ground from your Arduino to the relay. Next, choose one of the digital pins from your Arduino (doesn't matter which) and connect it to the relay control connections - this pin will control the opening and closing of your relay circuit. In my case, I had to use a ribbon cable as an adapter between the male pins on the relay and the male jumper wires. In the picture below, I've removed all of the LEDs and connected the relay (white wire is 5v power to relay, blue is ground, and red wire connects pin 10 to the 3rd relay control).

Now might be a good time to mount your 12v door latch. Once mounted, take the ground wire from the door latch and solder/connect it directly to the ground of your 12v power supply. Next, take the voltage-in wire from the door latch and connect it to the middle 'port' of one of your relays. This middle port is the 'common' connection, and it is always connected to the circuit. The other two 'ports' are normally open (NO) and normally closed (NC). Normally open means that the circuit is open (no current flowing through) when the relay is not powered; once the relay is powered by the 5v connection from the Arduino, the circuit closes and current flows through. The normally closed port is the opposite - current is flowing through until the relay gets a 5v signal, at which point it breaks (opens) the circuit. In our case, we want to connect the positive wire from the 12v power supply to the normally open contact. Here's a quick diagram to help:

We've just covered a lot so here's a quick recap video to bring it all together:

Alright, if you've managed to get all of the hardware put together, all you need is a little more code to interact with the relay and door latch. Here's my code:

/*####################################################################
tiltToUnlock
Alex Glover
February 2013
Combination to unlock box is Y+ (left), X+ (forward), Y+ (left), X- (back), Y+ (forward)
Based on autoCalibration.ino, provided by virtuabotix
#######################################################################*/
#include "Accelerometer.h"
Accelerometer myAccelerometer = Accelerometer();
/*int yPosPin = 11;
int yNegPin = 13;
int xPosPin = 10;
int xNegPin = 9;
int zPin = 12;
We don't need any of these pins anymore, as we're not using LEDs*/
int currentMove = 1;
int resetTrigger = 0;
int relayPin = 10;
void setup()
{
  Serial.begin(9600);
  //Connect up the following pins and your power rail
  // SL GS 0G X Y Z
  myAccelerometer.begin(3, 4, 5, A0, A1, A2);
  /*pinMode(yPosPin, OUTPUT);
  pinMode(yNegPin, OUTPUT);
  pinMode(xPosPin, OUTPUT);
  pinMode(xNegPin, OUTPUT);
  pinMode(zPin, OUTPUT);
  We don't need any of these pins anymore, as we're not using LEDs*/
  pinMode(relayPin, OUTPUT);//Set the pin that will control the relay in OUTPUT mode
  digitalWrite(relayPin, HIGH);//Write a high voltage signal to the relay on startup - this will break the circuit
  //calibration performed below
  Serial.println("Please place the Accelerometer on a flatnlevel surface");
  delay(2000);//Give user 2 seconds to comply
  Serial.println("Calibration complete");
  myAccelerometer.calibrate();
}
void loop()
{
  delay(20);//delay for readability
  resetTrigger++; //increment the reset trigger
  myAccelerometer.read();
  if(myAccelerometer._Zgs -0.9){
    //digitalWrite(zPin, LOW);
  }
  if(myAccelerometer._Xgs >= 0.5){
    //digitalWrite(xPosPin, HIGH);
    if(currentMove==2){
      Serial.println("Second move successful");
      currentMove++;
      delay(1000);
    }
    else if(currentMove==5){
      Serial.println("Fifth move successful - opening");
      currentMove=1;
      digitalWrite(relayPin, LOW);
      delay(2000);
      digitalWrite(relayPin, HIGH);
      delay(1000);
    }
    else{
      Serial.println("WRONG MOVE!");
      currentMove=1;
    }
  }
  else if(myAccelerometer._Xgs < 0.5){
    //digitalWrite(xPosPin, LOW);
  }
  if(myAccelerometer._Xgs -0.5){
    //digitalWrite(xNegPin, LOW);
  }
  if(myAccelerometer._Ygs >= 0.5){
    //digitalWrite(yPosPin, HIGH);
    if(currentMove==1){
      Serial.println("First move successful");
      currentMove++;
      delay(1000);
    }
    else if(currentMove==3){
      Serial.println("Third move successful");
      currentMove++;
      delay(1000);
    }
    else{
      Serial.println("WRONG MOVE!");
      currentMove=1;
    }
  }
  else if(myAccelerometer._Ygs < 0.5){
    //digitalWrite(yPosPin, LOW);
  }
  if(myAccelerometer._Ygs -0.5){
    //digitalWrite(yNegPin, LOW);
  }
  if(resetTrigger==3000){ //do this every minute (3000 iterations * 0.02 seconds = 60 seconds)
    currentMove=1; //reset currentMove back to 1, so user can try to do the sequence again
    resetTrigger=0; //set the resetTrigger back to 0 so it can start another iteration to 3000
  }
}

Awesome. One thing I almost forgot to mention - the 12v door latch needs something to latch on. You may have to get creative with this part, depending on what kind of box you are using. In my case, the cigar box lid actually slid off (rather than opening on hinges) so all I had to do was carve a notch into the lid for the latch to catch on.

Now when I slide the lid of the box again, it pushes the latch down until it lines up with the notch and then *click*, it latches and locks the box closed.

Whew. Long write-up. If I missed anything or if you have any questions, feel free to comment and I'll be happy to help!

Update: By connecting both the Arduino and the door latch to the ground and power rails on the breadboard, I was able to do away with the 9v battery altogether. Now you don't have to worry about the battery dying and the box being locked forever.

WordPress: Twenty Eleven Child Theme with Galleria Slider

2013-01-28T15:00:44+00:00

OK - I've been spending the last few hours getting acquainted with the WordPress backend (that sounds more risque than I had intended) and making my first custom 'child' theme.

If you have an interest in making custom themes and you're just getting started, I've got a few very generic tips:

Don't underestimate how complex some of the themes and frameworks can be
Start by creating a child theme or modifying an existing theme
Check out this tutorial on PressCoders

The tutorial on PressCoders is very clear, so I won't re-invent the wheel and write a tutorial on making child themes. Instead, I wanted to share the child theme that I made. Basically, I've taken the Twenty Eleven theme and replaced the static header image with a dynamic image slider (using Galleria). Here's a quick screenshot:

Anyways, as simple as it is, I've learned a lot from the exercise. If you want to use the theme or take a look at the code, check out the GitHub repo.

GitHub Repo

Thanks for reading.

Arduino Project 3: Accelerometer Primer

2013-01-21T15:00:21+00:00

This project is more of a warm-up than a full-blown project. I wanted to get a feel for the basic workings of an accelerometer in preparation for a larger undertaking (no spoilers!). I borrowed this project idea from a forum about Arduino and accelerometers but can't remember which forum, so I can't credit back to the original poster - sorry about that.

Before we get to the how, let's take a look at the what. Basically we have an accelerometer and 5 LEDs mounted on a breadboard. Four of the LEDs are mounted to correspond with the X and Y accelerometer axes and one LED is in the middle. When the accelerometer is level, only the middle LED is illuminated. If you tilt the accelerometer to the side, the LED on that side is illuminated.

Remember the accelerometer measures force, not tilt - we can demonstrate this by quickly moving the accelerometer to one side, creating lateral force. The LED on the opposite side relative to the direction we move the board will illuminate - that is, if we move the board to the left, the LED on the right will illuminate.

Let's move on to the how. Here are the parts I used:

Virtuabotix MMA7361 Three Axis Accelerometer Module - $10
Arduino Uno R3 - $26
400-point Breadboard with Jumper Wires - $9
5mm LEDs with Resistors (5 Colors, Pack of 25) - $7

Total project cost of $52, plus you'll have some leftover jumper wires and LEDs.

Since this is just a simple primer, I won't go into great detail. Basically we need to wire up the accelerometer as outlined in the Arduino sketch and we need to wire up each LED to an output pin and ground. If you need help with the wiring, here's a rough diagram. Note - Fritzing didn't include a piece that was identical to my accelerometer, so I used two 5 pin components - they represent one piece. Also, the wire colors don't have any logical meaning, they match my real-world build.

Here's what it looks like up close.

Here are a couple more angles of the wiring:

As far as the programming goes, I started with the example on this page of the Virtuabotix wiki. Here's the my finished sktech:

/*####################################################################
accelerometerPrimer
Alex Glover
January 2013
Based on autoCalibration.ino, provided by virtuabotix
#######################################################################*/
#include "Accelerometer.h"
Accelerometer myAccelerometer = Accelerometer();
int yPosPin = 11;
int yNegPin = 13;
int xPosPin = 10;
int xNegPin = 9;
int zPin = 12;
void setup()
{
  Serial.begin(9600);
  //Connect up the following pins and your power rail
  // SL GS 0G X Y Z
  myAccelerometer.begin(3, 4, 5, A0, A1, A2);
  pinMode(yPosPin, OUTPUT);
  pinMode(yNegPin, OUTPUT);
  pinMode(xPosPin, OUTPUT);
  pinMode(xNegPin, OUTPUT);
  pinMode(zPin, OUTPUT);
  //calibration performed below
  Serial.println("Please place the Accelerometer on a flatnLevel surface");
  delay(2000);//Give user 2 seconds to comply
  myAccelerometer.calibrate();
}
void loop()
{
  delay(20);//delay for readability
  myAccelerometer.read();
  if(myAccelerometer._Zgs <= -0.9){ digitalWrite(zPin, HIGH); } else if(myAccelerometer._Zgs > -0.9){
    digitalWrite(zPin, LOW);
  }
  if(myAccelerometer._Xgs >= 0.5){
    digitalWrite(xPosPin, HIGH);
  }
  else if(myAccelerometer._Xgs < 0.5){
    digitalWrite(xPosPin, LOW);
  }
  if(myAccelerometer._Xgs <= -0.5){ digitalWrite(xNegPin, HIGH); } else if(myAccelerometer._Xgs > -0.5){
  digitalWrite(xNegPin, LOW);
  }
  if(myAccelerometer._Ygs >= 0.5){
  digitalWrite(yPosPin, HIGH);
  }
  else if(myAccelerometer._Ygs < 0.5){
    digitalWrite(yPosPin, LOW);
  }
  if(myAccelerometer._Ygs <= -0.5){
    digitalWrite(yNegPin, HIGH);
  }
  else if(myAccelerometer._Ygs > -0.5){
    digitalWrite(yNegPin, LOW);
  }
}

You can adjust the values to increase/decrease sensitivity. Hope you enjoyed the post, and I hope this helps you jumpstart your project with your accelerometer!

Bootable USB with UNetbootin

2013-01-14T15:26:08+00:00

OK, so you have a netbook or ultrabook with no optical drive. Or maybe your optical drive is dead. Or maybe you're smart enough to not use optical drives in general. Whatever the situation is, you want to run or install something from a bootable USB drive.

There are lots of solutions out there, but nothing beats the Universal Netboot Installer, or UNetbootin. UNetbootin has a simple GUI and never gives any grief. You can use your own ISO that you've downloaded or use one of the 42 ISOs UNetbootin provides, including BackTrack, CentOS, Ubuntu, as well as several rescue and recovery utilities.

Using UNetbootin is stupid easy. Grab a USB flash drive with enough storage to hold your ISO. Better yet - pick up a 16GB flash drive for $10 and it will be big enough to hold any ISO you could ever want. Got your USB flash drive? Let's get started.

Plug your USB flash drive into your computer. Note the drive assignment; in Windows, it will probably be D: or E:, or some other letter. For the Unix users, your drive designation will probably be sda, sdb, sdc... sdx. Remember the drive assignment.
Go to UNetbootin's page and download the correct version for your OS.
If you have a particular ISO you want to use, download it. Otherwise if you want to use one of the included ISOs, skip this step.
Open UNetbootin. If you want to use one of the ISOs they provide, check the "Distribution" radio button and choose an ISO from the dropdown
If you want to use your own ISO, click the "Diskimage" radio button, click the browse button " ... " and find your ISO that you downloaded.
Change the drive to correspond with your USB flash drive. If you can't find your drive, make sure it's mounted and then restart UNetbootin.
Click OK. If you chose a distribution provided by UNetbootin, it will take a fair amount of time to download the ISO before it can write to your USB drive. This is normal.
Once the USB has been formatted and the ISO written to it, UNetbootin will restart your computer in an attempt to boot from the USB drive. If it doesn't work, go into your BIOS settings and try changing the boot order.

Donezo. Hope you found this useful.

Arduino Project 2: The Audible Eye Part 2

2013-01-09T15:00:03+00:00

Alright, if you're following along from Part 1, you should have your headphones/speakers as well as your ultrasonic rangefinder wired up and operational. Next we have to write the code to marry these two devices.

The Code

/*
Alex Glover
Copyright December 2012
The Audible Eye - handheld device that utilizes an ultrasonic rangefinder to determine distance to whatever object the device is pointed at.
Feedback will be delivered to operator via sound - tone will increase pitch for shorter distances, no tones played for extreme distances.

Credit to:
YourDuino SKETCH UltraSonic Serial 1.0
terry@yourduino.com

Uses: Ultrasonic Library (Copy to Arduino Library folder)
http://iteadstudio.com/store/images/produce/Sensor/HCSR04/Ultrasonic.rar
*/

/* Libraries */
#include "Ultrasonic.h"
/* Pin Assignments */
//Pin designated for controlling the emission of ultrasonic 'pings'
#define TRIG_PIN 2
//Pin designated to 'listen' for ultrasonic 'pings'
#define ECHO_PIN 7
//Pin used to drive the audio output through the headphones
#define SOUND_PIN 12
/* Declare objects */
//Create an object of type 'Ultrasonic' and call it 'Sensor,' passing the trig pin and echo pin numbers into the constructor
Ultrasonic Sensor(TRIG_PIN, ECHO_PIN);
/*-----( Declare Variables )-----*/
//Variable to hold the tone value; this value will be calculated first, and then passed to the tone() function
int toneToPlay=0;

void setup()
{
  Serial.begin(9600);
  //Pring the following text to the serial monitor
  Serial.println("The Audible Eye");
  Serial.println("alexdglover.wordpress.com alexdglover@gmail.com");

}
void loop()
{
  delay(200); //20 millisend delay to let echos from room dissipate

  //The sensor occasionally returns 0 incorrectly. 0 would also give us a divide by zero error in the next line of code. To address these issues
  //we use this simple if statement. Basically, if a 0 is returned, we play no tones through the headphones.
  if (Sensor.Ranging(INC) != 0){
    //Calculating the tone to play as 1000 hertz divided by the number of inches. So if the device is one inch away from an obstacle, we play
    //a 1000 hertz tone. If the device is 10 inches away, it plays a 100 hertz tone. Anything greater than 20 inches reduces the tone to almost inaudible.
    toneToPlay=(1000/Sensor.Ranging(INC));
  }
  else{
    toneToPlay=(0);
  }
  //Here we take whatever value toneToPlay was calculated at and pass it into the tone() function. The parameter of 200 is the number of milliseconds to play
  //the tone. The loop takes about 200 milliseconds in total, so if the loop hangs or crashes, the tone won't keep getting played indefinitely.
  tone(SOUND_PIN,toneToPlay, 200);
}

Upload that sketch to your Arduino and test. If all's well, you should be hearing various tones coming out of your headphones depending on what's in front of your rangefinder.

The Finishing Touches

Now at this point, you could declare this project a success. The device works as intended. However, I wanted to give this project a few finishing touches that I usually skip over, so let's put it all inside an enclosure, make it portable (i.e. battery powered), and throw on a badass toggle switch for the fun of it.

First, let's wire up the toggle switch. Take one of your 9V battery adapters and cut both wires about halfway between the battery clip and the male adapter. Now take the battery clip side and strip the wires. Connect the red wire to the voltage-in connection on the switch and the black wire to the ground. Now take the male adapter piece and strip the wires. Connect the red wire to the voltage-out connection on the switch and connect the black wire to the same ground connection on the switch we used before. My wiring is quite ugly, but you should get something that looks like this:

Now for the enclosure. I was having a hard time finding a legitimate project enclosure of the right size, so I just used an old cardboard box. Cut out a small rectangle for the the rangefinder to protrude from at the front of your device. On a different side, cut out another hole (size will depend on what size of toggle switch you use). Once you have your holes cut, start installing your components. For my enclosure, I installed everything but the Arduino on one 'level' and used some packing foam to wedge things into place.

Then I installed the Arduino and connected the power adapter, leaving the headphones outside of the box as I closed the lid. Here's a final closeup:

That's it, hope you enjoyed the post!

Arduino Project 2: The Audible Eye Part 1

2013-01-07T14:35:16+00:00

The Audible Eye is a short proof-of-concept project that I came up with while looking for ways to experiment with some new gear. The idea of combining an ultrasonic rangefinder and some sort of signaling audio output jumped out to me - it would give you a depth perception, not unlike echo location that bats and dolphins use. As far as practical use, I believe it could be used by the visually impaired as a complement to a white cane, but not as a replacement.

Basically, the tone/pitch of the audio signal would get higher as the operator moves closer to a wall or object, indicating to the operator that they are getting closer. Similarly, as the operator moves away from the wall the tone will drop until it's almost inaudible.

Let me provide some context to make this more clear. Let's imagine our operator is blindfolded. If the operator pointed the device down an empty hallway for example, they would hear almost no tone, telling them it's safe to walk forward. As they approached a wall, the tone would increase. The operator would then scan around them with the device, find another path that was unobstructed, and could continue walking.

This video is all but unwatchable (quality is IMPRESSIVELY bad), but it will at least give you an idea of how it works. And of course I disassembled the project before I realized the video was botched.

Alright, let's open up the enclosure and start breaking down the project. The entire project can be broken down into just a few components. Here's a view inside of the enclosure.

Component List:

Arduino Uno
HC-SR04 Ultrasonic Rangefinder
Breadboard and jumper wires
9v Battery Adapter
3 Pin Toggle Switch
9v Battery (stop putting it on your tongue)
Headphones (or any other audio output device)

Building the project can be broken down into phases. Let's focus on the ultrasonic rangefinder first, and build from there.

Setting up the Ultrasonic Rangefinder

There are 4 pins on the HC-SR04 ultrasonic sensor module: voltage in (labeled "Vcc" on my module), ground, TRIG, and ECHO. The TRIG pin will control the transmission of ultrasound. The ECHO pin will be the receiver pin. Use your jumper wires to connect the ground on the module to the ground on the Arduino (red wire near the top of the above picture) and connect the voltage in pin to the +5v pin on the Arduion (white wire near the top of the above picture). Next, use jumper wires to connect the TRIG and ECHO pins to two of the digital pins on the Arduino. These will be the numbered pins on the Arduino that do NOT have the tilde (~) prefix. In my build, I used pin 2 for the TRIG connection and pin 7 for the ECHO connection. Here's a better picture:

OK we're all wired up! There are several libraries for interacting with ultrasonic rangefinders. The one I used is probably out of date, but you can download it here.

Note: If you're having issues with this library, edit Ultrasonic.cpp and Ultrasonic.h. Replace this line:

#include "WProgram.h"

with this:

#include "Arduino.h"

Once you've downloaded it, add the entire directory (should contain Ultrasonic.h, Ultrasonic.cpp, keywords.txt, and an examples directory) to your Arduino's libraries directory. I'm being lazy and have the Arduino install on my desktop, so my library directory is here:

C:\Users\Alex\Desktop\arduino-1.0.1\libraries\

Once that's done, open up the Arduino IDE. Click on the "Sketch" dropdown menu, click "Import Library" and verify that Ultrasonic is a listed library. If it's there, you're in good shape.

Visit http://arduino-info.wikispaces.com/UltraSonicDistance and check out the Arduino Sketch provided there. Edit the TRIG_PIN and ECHO_PIN variables to coincide with the pins you used (remember I used 2 and 7 in my case). Connect your Arduino via USB and upload the Sketch. Finally, open the serial monitor (Tools dropdown --> Serial Monitor). You should see readings in inches and centimeters being reported back to the serial monitor.

Awesome - this is our first building block.

Setting up the Audio Output

This step is super easy, unless you're me and you waste an hour getting stereo wires confused. If you're using a piezo-electric piece or a simple mono-speaker or headphones, there should be only 2 wires - a voltage in and a ground. Simple. Now, the scrap headphones I had were stereo headphones, which threw me off. If you have a stereo device, you'll likely have 2 separate insulated sections with 2 types of wiring each. In each insulated wire is one plain copper wire and one painted wire. The copper wires are both grounds and the painted wires are the right and left voltage-in connections. Since we're just doing simple mono output, go ahead and twist the two copper wires together and twist the two painted wires together. Next, connect the copper ground wires to one of the grounds on the Arduino. For the voltage-in wires, connect them to one of the PWM pins on the Arduino (I used pin 11). Here's a closeup of the headphone wiring:

Now to test! Conveniently, the Arduino IDE (at least version 1.01) includes a few audio output sketches. Go to File --> Examples -->2. Digial --> Tone Melody. Search the Sketch for the tone() function - change the pin number in each of these functions to coincide with the pin you connected your headphones/speaker to. Now upload the Sketch to your Arduino - if everything is working, you should hear some the melody of "Shave and a haircut, two bits."

Awesome - now we have two of our building blocks completed.

Cliffhanger! In part 2, we'll wire up the toggle switch and battery, as well as write the code to combine the rangefinder and the headphones to complete "The Audible Eye."

Monitoring Disks in RAID Array in Windows 7

2012-12-05T14:30:11+00:00

For many folks whose motherboards don't support RAID or just want a simple, easy to implement RAID solution, software RAID is often the best solution. In Windows 7, Microsoft finally got around to providing out-of-the-box software RAID capabilities (apparently you could do it in Windows XP, but not without some real leg work). Only problem is Microsoft didn't include a notification or alarm mechanism to deal with disk failures. The only notification is a passive entry in the Event Log, which is not obvious for basic users and not really convenient for anyone. This effectively defeats the purpose of RAID as a mechanism for data redundancy, as your disks could fail one by one until all of your data was unrecoverable, and you as the user wouldn't realize until it was too late.

Fortunately there are several free disk monitoring applications that can fill this void Microsoft left for us. I've chosen DiskCheckup for one important reason - it has the ability to send email notifications when there's an issue (what good is a desktop alert, if you're not on your desktop?). DiskCheckup uses the S.M.A.R.T values from your hard drives to look for potential issues and notifies you if they exceed pre-set thresholds. Downloading and installing DiskCheckup is quite easy, but there are a few important configuration steps I wanted to highlight:

After installing, open DiskCheckup. Note - by default, it doesn't start the application after installation or after a reboot. Something to keep in mind if you want it running all the time
Click Configuration in the bottom left
Choose the Send e-mail notification radio button and click Settings
Enter the email address where you want notifications sent in the top right and click Add. For outgoing mail server settings, use your ISPs SMTP relay information. For example, here's a good list of Time Warner Cable SMTP relay servers. With a little bit of Googling, you should be able to find your relay. With my ISPs SMTP relay, I do not have to provide credentials. Also note, this screen allows you to specify a port - don't bother trying to use SMTP relays with TLS enforced, DiskCheckup doesn't support it. You can also optionally add a "From" address if you want, although that may cause issues with your SMTP relay.

That's it - fire off a test email to make sure it works, and click OK! An email will be sent if any of the S.M.A.R.T. values or the temperature exceeds their respective thresholds. You now have a vigilant application looking after your hard drives!

Easily Create a Store Locator

2012-11-28T14:15:41+00:00

I had initially set out to build my own custom store locator (just for the learning experience) and post a how-to article for it. I quickly found that this has been done many times before, and Google has a very good tutorial if you want to build your own.

But what if you don't have a database behind your website? What if you don't have any IT staff to support a custom store locator? What if you need an easy solution?

Check out BatchGeo, a free (with ads) hosted store locator service. All you need is a spreadsheet of your locations that you want to map and whatever secondary information you want to display. BatchGeo takes care of mapping the addresses, so you don't have to determine the latitude and longitude values for each address (like in the Google tutorial). You can also group your locations arbitrarily, such as by location type by simply adding another column of data.

Once you supply your information and click "Map Now," the map will be generated for you to preview. If all is well, just click Save & Continue. You'll be asked to provide an email address and some basic information about the map you generated, including whether you want it to be public or unlisted. Then just click Save Map.

Finally, BatchGeo will email you the new map URL, as well as some basic iframe code to embed your map on your website. The map has great functionality, allowing you to search geographically (by city, zip code, whatever) or by secondary content (store name, group name, even phone number). Content display is great. Markers are lettered to correspond to the location listing below (as we've come to expect from Google maps). Markers are also color coded if you used grouping. Info windows cleanly display formatted secondary information such as web site and phone number.

If you want to get rid of the embedded advertisements, unfortunately it's going to cost you $99.00 per month. If you can bear the ads, it's an awesome free service.

Prevent SQL Injection Attacks in PHP Applications

2012-11-26T00:00:00+00:00

Be advised This post is quite old (26 Nov 2012) and any code may be out of date. Proceed with caution.

Ok, some quick background before we get started. Let's say you have a search bar in your PHP-based web site to help people find items, which ties back to your database. Behind your search bar is some code and a query, something like

$query = "SELECT * FROM ITEMS_TABLE WHERE ITEM_NAME LIKE" . $mySearchBarString . ";";

So a user provides some string, like 'Xbox,' and it finds all items with 'Xbox' in the name. Lovely. But what if the user enters this:

blah; SET @tables = NULL;
SELECT GROUP_CONCAT(table_schema, '.', table_name) INTO @tables FROM information_schema.tables;
SET @tables = CONCAT('DROP TABLE ', @tables);
PREPARE stmt1 FROM @tables;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;

For those who can't read SQL, here's the short version - an attacker just dropped ALL of your databases, in all schemas. Whoops. This is your face right now:

We're not going to let this happen to us, because losing and replacing that data seems like a lot of work, and we all have better things to do than restore backups and try to explain data loss to end users.

Let's review some options.

Escape Special Characters

Good news - this is a super easy method in PHP. Bad news - it doesn't protect against all SQL injection attacks. To implement, simply pass your users' input through the mysql_real_escape_string function and assign it to a new variable. Then use that variable in your query. For example, let's assume a user has posted some input from a form, which we'll grab with the $_POST[x] variable:

$userInput = mysql_real_escape_string($_POST[some_input]);
$query = "SELECT * FROM ITEMS_TABLE WHERE ITEM_NAME LIKE" . $userInput .";";

Simple! However, mysql_real_escape_string was created primarily to sanitize inputs, not give 100% protection against SQL injection attacks. Clever attackers can input HTML into your database, setting you up for cross-site scripting attacks. mysql_real_escape_string also doesn't escape the percentage sign (%), the wildcard in SQL, leaving you vulnerable.

Just to be clear - don't use myql_escape_string, as it is also vulnerable to multi-byte character attacks. At the very least, use mysql_real_escape_string.

MySQLi and Prepared Statements

This post is already getting long-winded, so I'll be brief. Using prepared statements is a best practice for a boatload of reasons. The advantage is that you convert your users' string input into a parameter object. It is 100% immune to SQL injection, because the input isn't concatenated to the query, it's treated as an object. You can use either MySQLi or PDO prepared statements, both are acceptable. Here's a quick example of using MySQLi and prepared statements from my scratch-space website:

<?php
//set database connection
$mysqli = new mysqli("domain.com","username","password","database_name");

//open database connection; if connection fails, echo the mysql error
if ($mysqli->connect_errno) {
  echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") " . $mysqli->connect_error;
}

//check if there is an HTTP POST variable; if one exists, then we need to do the insert; otherwise, don't insert anything
if (isset($_POST["orgName"])) {
  /* Prepared statement, stage 1: prepare */
  if (!($stmt = $mysqli->prepare("INSERT INTO appgen (orgName, motto, number, map, site, body) VALUES (?, ?, ?, ?, ?, ?)"))) {
    echo "Error while preparing statement";
  }
  /* Prepared statement, stage 2: bind and execute */
  if (!$stmt->bind_param("ssssss", $_POST[orgName], $_POST[motto], $_POST[number], $_POST[map], $_POST[site], $_POST[body])) {
    echo "Binding parameters failed: (" . $stmt->errno . ") " . $stmt->error;
  }

  if (!$stmt->execute()) {
    echo "Execute failed: (" . $stmt->errno . ") " . $stmt->error;
  }
  else{
    echo "<script>alert('Record added!');</script>";
  }
}
?>

I think most of the code is pretty straightforward, but one thing that tripped me up is the "bind_param" function. The first argument in quotes is actually the data types of all of the parameters you want to bind. Each letter, in sequential order, represents the datatype of the corresponding parameter. There are only four supported datatypes, i for integer, d for double, s for string, and b for BLOB. So, for example, if I wanted to prepare an insert statement to insert an integer primary key, a string name, a decimal price, and an image BLOB, my code would look something like this:

<?php
...
if (!($stmt = $mysqli->prepare("INSERT INTO table (id, name, price, image) VALUES (?, ?, ?, ?)"))) {
  echo "Error while preparing statement";
}
/* Prepared statement, stage 2: bind and execute */
if (!$stmt->bind_param("isdb", 1,"Xbox", 299.99, image_binary)) {
  echo "Binding parameters failed: (" . $stmt->errno . ") " . $stmt->error;
}
...
?>

Also notice that the number of question marks should be the same as the number of parameters you are binding. The documentation at php.net is pretty good, but not great. If you have any questions or issues with this, please feel free to leave a comment.

As always, hope this was helpful!

Easy Modal Overlays

2012-11-21T14:00:12+00:00

Be advised This post is quite old (21 Nov 2012) and any code may be out of date. Proceed with caution.

Modal overlays are those fancy popups, often used to display a full-sized image or some sort of form. They give the appearance of a new window within the browser, giving sites a more lively animated and interactive look.

It's a pretty sweet feature, and is actually pretty easy to implement. I'll highlight a couple options and talk about how to implement them in this post.

Lightbox

Lightbox is a well known modal overlay package that uses jQuery. The implementation of Lightbox is very straightforward

Download Lightbox files</li>
Move the files into their respective folders where your site is hosted(e.g. move all of the Lightbox CSS files to the directory where you host all your CSS files)</li>

Add script references to the head of your HTML document in the right order:

<script src="js/lightbox.js"></script>
<link href="css/lightbox.css" rel="stylesheet" />

Make sure that the path to the Lightbox images is correct in lightbox.css and screen.css files, otherwise the images won’t render</li>

Now for the best part - add your images using an anchor tag, like this:

<a href="img/image1.jpg" rel="lightbox" title="some caption">some arbitrary text</a>

That's it! The CSS and Javascript take care of the rest! Each image can have a title (just modify the title attribute). You can also have a gallery of images, by using the same group name in each rel attribute, like this:

<a href="img/image2.jpg" rel="lightbox[group_name]">Image 2 Name</a>
<a href="img/image3.jpg" rel="lightbox[group_name]">Image 3 Name</a>

Custom (but easy) Modal Overlay with jQuery Tools

If you want to do something a little different than Lightbox, check out the jQuery Tools Overlay. Also built using jQuery, the jQuery Tools approach is a little less prescribed, a little more open.

Here's the step-by-step approach:

Add images to your web site or web app with img tags, giving an arbitrary value to the rel attribute with a # sign like this:
```
<img src="img/image2.jpg" rel="#arbitrary2"/>
```
Create the content you want to appear in the modal overlay within a div tag. Assign the same arbitrary value to the id attribute, and give the div a class name that you will create in the CSS like this:
```
<div class="arbitrary_class_name" id="arbitrary1">...</div>
```

Define the “arbitrary_class_name” in your CSS file. I re-used most of the CSS on the jQuery Tools’ tutorial:

.arbitrary_class_name
{

/* must be initially hidden */
display:none;

/* place overlay on top of other elements */
z-index:10000;

/* styling */
background-color:#333;

width:675px;
min-height:200px;
border:1px solid #666;

/* CSS3 styling for latest browsers */
-moz-box-shadow:0 0 90px 5px #000;
-webkit-box-shadow: 0 0 90px #000;
}
/* close button positioned on upper right corner */
.
arbitrary_class_name .close {
background-image:url(img/close.png);
position:absolute;
right:-15px;
top:-15px;
cursor:pointer;
height:35px;
width:35px;
}

Don’t forget to add a reference to your CSS file in the head of the HTML file, as well as a reference to the jQuery Tools javascript:
```
<link href="css/yourStyleSheet.css" rel="stylesheet" />
```

Last step - add this short Javascript to the your HTML document:

$(document).ready(function() {
  $("img[rel]").overlay();
  });
</script>

That's it!

Making Your Site Mobile-Friendly

2012-11-19T14:02:05+00:00

Be advised This post is quite old (19 Nov 2012) and any code/references may be out of date. Proceed with caution.

There's no refuting the statistics - over 20% of all web traffic comes from mobile users, and almost everyone agrees that this figure will grow.

Mobile users can use standard web sites, but mobile-friendly sites are going to have better click through, more page visits, and more time spent on pages (there's probably a cooler digital marketing term for this, but you get the idea).

Part of the pain of mobile-friendly sites is that you must accommodate many different screen sizes and aspect ratios. Check out this comparison from Phone Size:

Very different from one to the next. And realistically, this is a very small sampling of phone sizes.

Let's walk though a couple scenarios and talk about solutions.

Starting From Scratch

If you have no web presence at all, you have the advantage of a clean slate. You don't have to try to retrofit your existing content and code to work on various screen sizes. If you have no technical background, plan on hiring a web developer or contracting out to a firm. If you are like me and can code well enough to get yourself into trouble, check out initializr, specifically the Responsive and Bootstrap package demos. initialzr provides frameworks for building mobile-friendly sites - if you are on your computer, try re-sizing your browser window so it's about the size of a smart phone screen. You'll see that the layout of the page will actually change based on the window size. This is what's called a 'responsive layout' or 'responsive web design.' The advantage is you build your site once, and it should work on any device. Obviously, as you customize the frameworks and add your own content, you have to be careful not to violate the framework such that it can no longer adapt to the window size.

Re-directing to a Mobile-Friendly Page

This approach is a very sharp double-edged sword, depending on how you do it. The advantage is clear - you don't have to butcher your existing site to create a mobile friendly page.

Re-directing browsers to different pages based on their browser is quite easy. It is even easier if you hop over to Detect Mobile Browsers' web site and have them generate the code for you! I've used their Javascript code for a redirect in the past, and it was dirt simple. I've only used/tested their Javascript code, and it didn't actually work out-of-the-box. When I deployed their code, it redirected mobile browsers to THEIR site. I had to add the URL to my mobile site to the code in place of the default value of their URL. Just a heads up, you may need/want to double check their code. Otherwise it works great!

The disadvantage of using a redirect - be prepared to double the amount of web work you do going forward. You now have two totally disparate sites to maintain. You may be able to re-use some of your CSS styles from your original page, but unless you get clever about how you populate the content of your sites, you could be doing a lot of duplicate work. Look for opportunities to use server-side code to inject your text/content into pre-defined areas in both sites to limit the impact.

Retrofit an Existing Site to be Mobile-friendly

Doable, but obviously this is going to heavily depend on how your site is structured now, so I can't give you many specifics. Stick with percentages for sizing components, avoid fixed pixel counts. When dealing with fonts, try using em for sizing, so that size is always relative. Keep in mind that mobile browsers have two views, landscape and portrait, which may vary heavily by device (both in raw value and aspect ratios). For forms, try to stick with standard HTML input tags, as all mobile devices can handle them gracefully.

Hope some of those thoughts and resources help, and please post comments if you have any questions.

Konami Code

2012-11-16T13:30:38+00:00

The Konami code... if you were born in 1990 or earlier, and play (or played) video games, you probably already know what I'm talking about.

The history of the Konami code is awesome in it's own rite. Here's the short version: the Konami code is a famous cheat code (up, up, down, down, left, right, left, right, B, A) that first appeared in NES games like Contra. Many times the code was added to video games as a tribute to the classic games or as a tribute to Kazuhisa Hashimoto, the developer who accidentally left the cheat code in the game when it was sold to the public.

That homage has been proliferated by nerds and techies through many games and, eventually, made it's way on websites.

YES, THERE IS A CHEAT CODE FOR THE INTERNET.

Sort of. It usually just reveals an easter egg on that site. You can find a whole list of websites that use Konami codes at Konami Code Sites. It's a lot of fun finding the secret content people have put behind respectable websites.

Some of you are probably thinking "But how do I put a Konami code on MY site?!?!" Well, at least that's what I was wondering.

There is a pretty robust Konami code javascript library that's even iPhone compatible, hosted at Google Code (on an iPhone, it's swiping and tapping, so it's "up, up, down, down, left, right, left, right, tap, tap, tap".

For those who are looking for the quickest and easiest version, I found this over at snaptortoise.com:

<script type="text/javascript" src="http://konami-js.googlecode.com/svn/trunk/konami.js"></script>
<script type="text/javascript">// <![CDATA[
konami = new Konami()
konami.load("http://www.google.com");
// ]]></script>

But that… is almost too easy. And you may want to do more custom things than just loading a different page. Or maybe you want to do a slightly different cheat code. I found some simpler, flexible (albeit not as flashy) javascript code on another blog:</p>

// check to make sure that the browser can handle window.addEventListener
if (window.addEventListener) {
  // create the keys and konami variables
  var keys = [],
  konami = "38,38,40,40,37,39,37,39,66,65";
  // bind the keydown event to the Konami function
  window.addEventListener("keydown", function(e){
    // push the keycode to the 'keys' array
    keys.push(e.keyCode);
    // and check to see if the user has entered the Konami code
    if (keys.toString().indexOf(konami) >= 0) {
      // do something such as:
      alert('Konami');
      // and finally clean up the keys array
      keys = [];
    };
  }, true);
};

The "38,38,40,40,37..." are keypress IDs, so you would change this portion if you wanted a different cheat code.

There's even a WordPress plugin for Konami codes!

Hope this was helpful, and enjoy the easter egg hunt!

Google Hacking

2012-11-14T13:00:53+00:00

When I was still working at the University of Wisconsin, a co-worker of mine showed me a way to find unprotected directories indexed by Apache and web cameras. Not those kind of web cam feeds - these are usually security cameras in innocuous places, like restaurants, streets, tops of buildings... etc., that are so benign (usually) that whoever set them up decided they didn't need to be secured (or didn't know how to secure them).

Anyway, the way he found these was the interesting part. At the time, it was termed "Google Hacking" although now it is sometimes called "Google dorking." It is simply using Google to find unsecured web content with targeted search terms. You're generally looking for things like:

Content that has been indexed by Apache or IIS (or any webserver for that matter), but isn't secured. This allows you to find web content that may not be intended for public consumption (i.e. there aren't links from web pages to these files). A good example would be to Google this:
intitle:index.of "Apache/2.0 Server at"
This will yield a LOT of results, most of which will be normal and of no value. Additional operators/terms would be needed to find anything interesting very quickly
Password files of various types. This seems silly, but there are a lot of password files that are on the internet. The most vulnerable are clear-text passwords (as opposed to hashed passwords). Once you have someone else's username and password, you proceed to do your malevolent activities. A good example of a Google search for clear-text passwords is:
filetype:log inurl:"password.log"
Sensitive data such as Personally Identifiable Information (PII) or credit card information, that is stored in clear-text
FTP Sites that allows anonymous connections

You can actually see a lot of the world through Google hacking links to webcams. You can try this one for example:
intitle:”EvoCam” inurl:”webcam.html”

The first result, for example, is a camera over an outdoor eating area at The Salty Dog Cafe.

Or you can find network printers that are accessible to the world wide web, and print funny pictures on their printer. You can find a fair number of printers with these search terms:
inurl:”printer/main.html” intext:”settings”

Maybe you could print this for them:

You can find a lot of information on dorking and some good examples at the Google Hacking DataBase.

One last note - if you're doing this for malevolent purposes, you may get 'caught' - IT security folks are not dumb, so if you try to malevolently attack an organization this way, you are most likely going to find a 'honeypot.' A honeypot is an INTENTIONALLY CREATED VULNERABLE AREA to draw in rookie hackers and bust 'em or at the very least, log the malevolent attempt. If you're doing this for learning/academic worries, I wouldn't sweat it. On the other hand, should you be taking any legal advice from me?

Posting Source Code in Wordpress

2012-11-12T15:00:41+00:00

This one is pretty much a no-brainer, but I wasn't aware of it until very recently so I thought I'd share.

Source code, as you've probably seen on lots of web pages, is treated special. And for good reasons, like readability, syntax highlighting, and to protect against the use of formatted instead of plain-text characters.

When I first started using my Wordpress site, I just used the <code> HTML tag. Which is fine. But there is no syntax highlighting, which makes reading code a little more painful. I stumbled on the Syntax Highlighter plugin which does a great job and covers many languages, but alas - I'm hosted on Wordpress.com and am restricted from adding my own plugins.

But wait - what's this? Wordpress.com has integrated Alex Gorbatchev's Syntax Highlighter into their instances! So all I need to do is switch from the <code> tag to the [sourcecode language="language"] tag! Boom, syntax highlighting in my Wordpress posts.

Here's a quick example of what the syntax highlighting looks like for HTML, just for kicks:

Note The snippet below was supposed to demonstrate the WordPress syntax plugin, but this site is now hosted on Jekyll. Do not expect the result below to resemble WordPress’s syntax highlighting.

<html>
<head>
<title>This is my web page title - notice it is not highlighted by the syntax highlighter, but the tags are</title>
</head>
<body>
Awesome.
</body>
</html>

This was a game changer for me, to the point that I've been editing my old posts to use this new tag so I can offer syntax highlighting to my readers.

To see a full list of the supported languages, check out the official Wordpress post about posting source code.

Update: I got burned by the WordPress auto-save feature, so this post might not have rendered correctly before. Working now!
Update of the update: It wasn't the auto-save that burned me, it was the sourcecode tag. It has a nasty tendency to convert symbols into their ascii codes (like "&gt" instead of ">"). Real inconvenience, sort of ruins all of the benefits of the syntax highlighting and the like. Anyway, just be aware when you are using it.

Deleting Files Based on System Timestamp

2012-11-10T14:07:08+00:00

Recently ran into an issue at work where a single database was creating A LOT of trace files. 54 gigabytes of trace files to be specific. Well, this was a problem because the system disk was only 56GB, preventing me from building new databases. No problem, its a dev environment, I'll just delete the trace files that are older than 3 days. Except that the trace file naming convention wasn't consistent, so there was no way for me to delete the oldest files based only on file name.

What about the system timestamp on each file? It seemed like a good solution, but I didn't want to write a shell script to do something so minor. Then I got lucky on Google, and found this useful little gem:

find /full/path/to/directory -type f -mtime +3 -exec rm -f {} ;

I'll translate - it will find all files in a given directory that have system timestamps from 3 days ago or greater. Clarifying note - the system timestamp reflects when a file was created OR when it was last modified. For each file that it finds that matches that criteria, it will execute 'rm -f {file name}' and delete the file!

You can either use the full path to the directory you want to delete files in, or if you're already in that directory, simply enter a single period, like this:

find . -type f -mtime +3 -exec rm -f {} ;

In my case, I wanted to delete all files older than 3 days. If you wanted to keep 30 days worth of trace files, you would just enter +30 instead.

find . -type f -mtime +30 -exec rm -f {} ;

Back to my example. I executed the above command and then… nothing. The Unix command line is often an unforgiving wasteland of lack of information. As I was deleting hundreds of thousands of trace files, the system hung for some time, and I couldn’t tell what kind of progress, if any, was being made. Fortunately the command was working, it just wasn’t obvious to me.

Luckily, the Unix ‘find’ binary allows you to call multiple command executions for each file that is found. To do so, terminate your first execution statement (the commands that follow ‘-exec’) with ‘;’ and add another -exec statement. Now, if we simply add another execution statement that calls a command to print a period (or any other string), we’ll be able to see the progress in the terminal. The printf binary will work quite nicely for this:

find . -type f -mtime +3 -exec rm -f {} ; -exec printf "." ;

One final tip. When using find-exec, you will occasionally get this error: find: missing argument to '-exec'

This often means that the -exec statement is processing the semi-colon as part of the exec command, rather than terminating it. You can resolve this by escaping the semi-colon with a backslash:

find . -type f -mtime +3 -exec rm -f {} ; -exec printf "." \;

Hope this helps!

Credit to Just Skins web development forum and UnstableMe for getting me started.

Simple Page-of-Posts Solution for Wordpress

2012-11-04T14:44:47+00:00

Be advised This post is quite old (04 Nov 2012) and any code may be out of date. Proceed with caution.

Recently I have been helping Make a Difference Wisconsin, a local non-profit, with their Wordpress website. Without going into details as to why, I needed to create a simple 'page of posts' for menu hierarchy reasons. It needed to be a page, but I wanted it to display all of the posts from a particular category. Google took me to many solutions, including the Wordpress Codex, but none of them worked. Some used out of date or deprecated query functions and some didn't work with the custom theme that was already in place. This should have been a very easy task, but took me several days to pin down my solution.

So, if you want to create a simple page-of-posts that should work in any theme (with some minor tweaking, of course), feel free to use this code:

<?php
/*
Template Name: pageofposts
*/
/*
Author: Alex Glover
Date: 4-Nov-2012
*/
?>
<?php get_header(); ?>
<div id="content-area">
<?php
global $post;
$args = array( 'numberposts' => 5, 'category' => 47 );//CHANGE THESE VALUES AS NEEDED
$myposts = get_posts( $args );
foreach( $myposts as $post ) : setup_postdata($post); ?>
<h2><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h2><br />
<?php if ( has_post_thumbnail()) : ?>
<a href="<?php the_permalink(); ?>" title="<?php the_title_attribute(); ?> " style="float:left; position: relative; margin: 0px 20px 0px 0px;">
<?php the_post_thumbnail('thumbnail'); ?>
</a><p style="width:450px; float:left; position: relative;">
<?php the_excerpt(); ?>
</p>
<br /><br /><br />
<?php endif; ?>
<?php endforeach; wp_reset_postdata(); ?>
<div></div>

</div> <!-- #content-area -->

<?php get_sidebar(); ?>
<?php get_footer(); ?>

Here's a quick preview of what the 'page of posts' code looks like, but obviously aspects of this will be very different based on the theme that you are using.

Automate remote password changes with BASH and 'Expect'

2012-09-15T20:13:41+00:00

I think this topic has been covered, in varying levels, by many people on the internet. Still, I felt like I was bashing my head against the wall trying to figure this out. To prevent head trauma to some other sys admins, I thought I'd post this.

First, some assumptions. Let's assume you're a sysadmin, working mainly in a LAMP environment (or at least Linux and MySQL). Let's also assume you have a reasonable number of hosts, between 5 and 50. You don't really have the right scale to warrant setting up RADIUS or TACACS authentication systems, but it is a real pain in the ass to reset passwords on a regular basis. Last, let's assume that for some good reason, you use the same password across many machines, both for the OS and DB authentication. To change all of those passwords, you would have to SSH into each box, run a couple password change commands, and exit each host.

Using BASH and 'Expect,' I've automated that process. For my experiment, I assumed that I wanted to change OS passwords for users root, oracle, and asm and DB passwords for root and oracle. I provide a list of hosts that are going to be updated in a control file titled 'password_change_list' There is a single shell script, 'change_password' (to invoke, just cd to the appropriate directory and type ./change_password'). Instead of taking a bunch of screenshots, I've included a quick screencapture of the script in action!

In the demo, I show the contents of the password_change_list control file, 2 hosts - localhost and my dev box. Add more hosts to this file as necessary. I start the script, which first asks you to confirm that you updated the control file. Next it asks for the current password, and then for the new password. The script will then SSH to the first host and run the password change commands, then exit. The script then loops and SSH's to the next host, and so on (in the demo, you'll see it fails to log on to my dev box via SSH, incorrect password). Now, since we're entering passwords to the command line and logging the commands, the passwords will be in clear text. To address that, the script will ask if you want to obfuscate all of the clear text passwords in the log. If you choose no, it will warn you again that the passwords are in clear text. Alternatively, if you leave clear text passwords in the log file, you can run the 'obfuscate_passwords' script, provide the new password, and it will replace all instances of the password with asterisks.

An archive can be downloaded here. Otherwise you can check out the scripts in GitHub:

GitHub repo

Obviously, the script will need some tweaking to suit your purpose, but I hope this helps you get started.

Arduino Project 1: Heat Seeking Robot (Part 1.5)

2012-08-23T03:25:25+00:00

Be advised This post is quite old (23 Aug 2012) and any code may be out of date. Proceed with caution.

I've procrastinated on this project post far too long! I've had the Heat Seeking Robot working at a very primitive level for over a month now, and posted NOTHING. For this, I am sorry.

So this is Part 1.5, where Part 2 was supposed to be the ultimate conclusion to the project. Here's where we're at so far: we have a working IR Thermometer reporting temperature readings to an Arduino board from Part 1. From there, I removed the IR thermometer from the breadboard, and pushed the four pins through one of the servo armatures (the white plus sign shaped object on top of the blue servo). As a throw-away proof-of-concept build, I just used double sided tape to attach the servo to the bread board (this will make more sense later).

Servo and IR Thermometer Mounted on Breadboard

I then pushed the jumper wires into the same servo armatures, originally intending to solder the wires, but it's actually a pretty good connection! I wired up the servo and then combined the IR thermometer sketch with the example sketch 'sweep.' Sweep is a simple servo sketch that makes a servo turn back and forth. Now I could get temperature readings as the thermometer turned back and forth. Not bad so far.

I assembled the Tamiya Tracked Chassis and the double gearbox (with a 37.5x1 gear ratio). The Tamiya chassis kit includes a battery mount for two AA batteries, but I used two 9v batteries instead for space-saving reasons. The battery mount is mounted in the middle of the chassis, which uses up a fair amount of space. By using 2 9v batteries sitting side by side, I got a simple and efficient way to have two mobile power supplies. I also attached some jumper wires to the two motors (two wires per motor, ground and positive; see the white and black wires on the top motor, blue and orange wires on the bottom motor).

Assembled Tamiya Tracked Vehicle Chassis

Attached the DF Robot motor shield to the Arduino board. Keep in mind that our connections for the IR thermometer from Part 1 will have to plugged into the corresponding connections on the motor shield. The motor shield transparently connects to the Arduino for all of the analog slots, as well as the 5v, 3v3, and GND connections on the DC connection side of the board (as opposed to the ethernet connection side of the board). Black circuit board is the motor shield, blue board is the Arduino. The pink foam is some packaging foam I got when I ordered these supplies that I use to make sure the board doesn't get shorted when touching the metal battery exterior. May not be necessary, but hey, better safe than sorry.

Arduno and Motor Shield

As you can see in the above picture, we're using mostly the same pins, just on the motor shield now and not directly on the Arduino. One thing I should have mentioned much much earlier: generally, when you have motors or other devices that consume a fair amount of power (relative to the Arduino), you will want two (or more) separate power sources, one specifically for the Arduino, and separate power sources for the motors. I used one of the male DC plug to 9v battery clips I bought to connect the Arduino and one of the 9v batteries. I used the other 9v battery to power the two DC motors. See the picture below; you'll need to connect your secondary power source (for the motors in this case) to the power-in connections you see on the right. Similarly, each AC motor needs to have a V-in and GND connected to the two green connection blocks in the middle of the picture.

Motor Shield Power and Motor Connections

Alright, so we've got all of the building blocks in place, let's assemble and see what we've got.

Heat Seeking Robot in Progress

Alright, so I attached the bread board to the top of the gear box, again by cheating and using double sided tape. The Arduino/motor shield is sitting on top of the two 9v batteries, which I strapped down using a velcro strap (normally used for cable management).

Fast forward through some serious trial and error on writing the application, and voila! We've got... a robot that sort of heat seeks. The idea is simple - the IR thermometer moves back and forth, scanning a room for humans (or anything warmer than 80 degrees fahrenheit). It records the direction of the heat source relative to the direction the robot is facing. The robot should take that direction and adjust until the object is straight ahead of the robot, then drive forward. Heat seeking, at a primitive level.

My robot currently works, except that it has a fixed turn for course correction. In short, if you're 45 degrees left of the robot, the robot will turn left (as intended!) but too far left. So then the target is to the right, and it turns right (as intended!) but too far right. Since it is a static course correction, the robot never lines up perfectly with the target. Getting so close to completing the project took some of the wind out of my sails. So I hope you enjoyed this as an interim post, and I promise to finish the robot and post the sketch as well. Please comment, or feel free to share your ideas for improvements!

Here's the current (incomplete and very unpolished) version of the sketch:

// A lot of this code is borrowed from geniuses. I'm crediting all of them up top, as they should be.
// ***
// Sweep
// by BARRAGAN <http://barraganstudio.com>
// This example code is in the public domain.
// ***
// ***
// MLX90614 IR Thermometer code taken from http://bildr.org/2011/02/mlx90614-arduino/
// ***
// Instructions for wiring thermometer here: http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1214872633
// RGB LED wiring and simple fading sketch: http://wiring.org.co/learning/basics/rgbled.html
// RGB+Arduino Project found here: http://publiclaboratory.org/es/node/593
#include <Servo.h>
#include <i2cmaster.h>
#include <AFMotor.h></code>
AF_DCMotor rMotor(3); // for M2 on Adafruit Motor Shield - not sure why
AF_DCMotor lMotor(4); // for M1 on Adafruit Motor Shield - not sure why
Servo myservo; // create servo object to control a servo
int pos = 0; // variable to store the servo position
int courseChange = 0; // variable to hold the value of how much of a course correction is needed - always less than 90
// set your thresholds for temperatures here - PROBABLY WON'T NEED ALL OF THESE IN MY APPLICATION
int cold = 60;
int normal = 80;
int hot = 100;
String tempLabel = " degrees F";
void setup()
{
myservo.attach(7); // attaches the servo on pin 9 to the servo object
// turn on motors
//lMotor.setSpeed(100);
lMotor.run(RELEASE);
//rMotor.setSpeed(100);
rMotor.run(RELEASE);
Serial.begin(9600); // Begin serial output at 9600 BAUD
Serial.println("Setup..."); // Print a quick note to terminal
i2c_init(); //Initialise the i2c bus
PORTC = (1 << PORTC4) | (1 << PORTC5);//enable pullups
}
void loop()
{
int highestTemp = 0; // integer to hold the value of the highest temperature reading so far - will be
int pointOfInterestCount = 0; //integer to hold the value of points of intersest; will divide total heading to get average heading
int pointOfInterest = 0; // integer, between 0 and 180, to indicate which direction the 'point of interest' is
int totalPointOfInterest = 0; //incremented integer that is a sum of all headings; will be divided to get average heading
for(pos = 10; pos < 180; pos += 1) // goes from 0 degrees to 180 degrees in steps of 1 degree
{
// Insane IR Thermometer handling code - I do not pretend to understand this code.
int dev = 0x5A<<1;
int data_low = 0;
int data_high = 0;
int pec = 0;
i2c_start_wait(dev+I2C_WRITE);
i2c_write(0x07);
// END Insane IR Thermometer handling code
myservo.write(pos); // tell servo to go to position in variable 'pos'
// IR Thermometer takes a reading
i2c_rep_start(dev+I2C_READ);
data_low = i2c_readAck(); //Read 1 byte and then send ack
data_high = i2c_readAck(); //Read 1 byte and then send ack
pec = i2c_readNak();
i2c_stop();
//This converts high and low bytes together and processes temperature, MSB is a error bit and is ignored for temps
double tempFactor = 0.02; // 0.02 degrees per LSB (measurement resolution of the MLX90614)
double tempData = 0x0000; // zero out the data
int frac; // data past the decimal point
// This masks off the error bit of the high byte, then moves it left 8 bits and adds the low byte.
tempData = (double)(((data_high && 0x007F) << 8) + data_low); tempData = (tempData * tempFactor)-0.01; float celcius = tempData - 273.15; float fahrenheit = (celcius*1.8) + 32; if(fahrenheit > normal){ // If the IR thermometer gets a reading greater than 60F
//Serial.println("above 80F"); // print to console for reference
if(fahrenheit > (highestTemp+1)){ // If the IR thermometer gets a reading that is the highest reading so far
//Serial.println("above 80F and the new highestTemp at"); // print to console for reference
highestTemp = fahrenheit; // Set the 'highestTemp' variable to the new temperature
pointOfInterest = pos; // Record the 'heading' that the servo is currently pointing
//Serial.println(pointOfInterest);
//Serial.println("degrees");
}
}
else if (fahrenheit <= cold){
Serial.println("below 60F"); // print to console for reference
}
delay(5); // waits 50ms for the servo to reach the position and for the IR thermometer to get a good reading
}
Serial.println(highestTemp + tempLabel);
Serial.println(pointOfInterest);
uint8_t i;
courseChange = 0;
if((pointOfInterest < 80) && (pointOfInterest != 0)){ /*courseChange = 90 - pointOfInterest; i = courseChange;*/ rMotor.run(FORWARD); Serial.println("to the left"); rMotor.setSpeed(255); delay(1000); rMotor.setSpeed(0); } else if(pointOfInterest > 100){
/*courseChange = 90 - pointOfInterest;
i = courseChange;*/
lMotor.run(FORWARD);
Serial.println("to the right");
lMotor.setSpeed(255);
delay(1000);
lMotor.setSpeed(0);
}
else if((pointOfInterest >= 80) && (pointOfInterest<=100)){ lMotor.run(FORWARD); rMotor.run(FORWARD); Serial.println("straight ahead"); lMotor.setSpeed(255); rMotor.setSpeed(255); delay(2000); lMotor.setSpeed(0); rMotor.setSpeed(0); /*digitalWrite(4,HIGH); digitalWrite(7, HIGH); analogWrite(5, 255); //PWM Speed Control analogWrite(6, 255); //PWM Speed Control delay(2000); analogWrite(5, 0); //PWM Speed Control analogWrite(6, 0); //PWM Speed Control */ } else if(pointOfInterest == 0){ Serial.println("No target - turning 90 degrees left"); rMotor.run(FORWARD); rMotor.setSpeed(255); delay(1000); rMotor.setSpeed(0); } delay(1000); for(pos = 180; pos>=10; pos-=1) // goes from 180 degrees to 0 degrees in steps of 1 degree
{
// Insane IR Thermometer handling code - I do not pretend to understand this code.
int dev = 0x5A<<1;
int data_low = 0;
int data_high = 0;
int pec = 0;
i2c_start_wait(dev+I2C_WRITE);
i2c_write(0x07);
// END Insane IR Thermometer handling code
myservo.write(pos); // tell servo to go to position in variable 'pos'
// IR Thermometer takes a reading
i2c_rep_start(dev+I2C_READ);
data_low = i2c_readAck(); //Read 1 byte and then send ack
data_high = i2c_readAck(); //Read 1 byte and then send ack
pec = i2c_readNak();
i2c_stop();
//This converts high and low bytes together and processes temperature, MSB is a error bit and is ignored for temps
double tempFactor = 0.02; // 0.02 degrees per LSB (measurement resolution of the MLX90614)
double tempData = 0x0000; // zero out the data
int frac; // data past the decimal point
// This masks off the error bit of the high byte, then moves it left 8 bits and adds the low byte.
tempData = (double)(((data_high && 0x007F) << 8) + data_low); tempData = (tempData * tempFactor)-0.01; float celcius = tempData - 273.15; float fahrenheit = (celcius*1.8) + 32; if(fahrenheit > normal){ // If the IR thermometer gets a reading greater than 60F
//Serial.println("above 80F"); // print to console for reference
if(fahrenheit > (highestTemp+1)){ // If the IR thermometer gets a reading that is the highest reading so far
//Serial.println("above 80F and the new highestTemp at"); // print to console for reference
highestTemp = fahrenheit; // Set the 'highestTemp' variable to the new temperature
pointOfInterest = pos; // Record the 'heading' that the servo is currently pointing
//Serial.println(pointOfInterest);
//Serial.println("degrees");
}
}
else if (fahrenheit <= cold){
Serial.println("below 60F"); // print to console for reference
}
delay(5); // waits 50ms for the servo to reach the position and for the IR thermometer to get a good reading
}
Serial.println(highestTemp + tempLabel);
Serial.println(pointOfInterest);
i = 0;
courseChange = 0;
if((pointOfInterest < 80) && (pointOfInterest != 0)){ /*courseChange = 90 - pointOfInterest; i = courseChange;*/ rMotor.run(FORWARD); Serial.println("to the left"); rMotor.setSpeed(255); delay(1000); rMotor.setSpeed(0); } else if(pointOfInterest > 100){
/*courseChange = 90 - pointOfInterest;
i = courseChange;*/
lMotor.run(FORWARD);
Serial.println("to the right");
lMotor.setSpeed(255);
delay(1000);
lMotor.setSpeed(0);
}
else if((pointOfInterest >= 80) && (pointOfInterest<=100)){
lMotor.run(FORWARD);
rMotor.run(FORWARD);
Serial.println("straight ahead");
lMotor.setSpeed(255);
rMotor.setSpeed(255);
delay(2000);
lMotor.setSpeed(0);
rMotor.setSpeed(0);
/*digitalWrite(4,LOW);
digitalWrite(7, LOW);
analogWrite(5, 255); //PWM Speed Control
analogWrite(6, 255); //PWM Speed Control
delay(2000);
analogWrite(5, 0); //PWM Speed Control
analogWrite(6, 0); //PWM Speed Control */
}
else if(pointOfInterest == 0){
Serial.println("No target - turning 90 degrees left");
rMotor.run(FORWARD);
rMotor.setSpeed(255);
delay(1000);
rMotor.setSpeed(0);
}
delay(1000);
}

Arduino Project 1: Heat Seeking Robot (Part 1)

2012-07-21T00:53:00+00:00

Be advised This post is quite old (21 Jul 2012) and any code may be out of date. Proceed with caution.

After finishing a cheerlights project for the GE Healthcare Tech Conference back in April, I had a pretty strong interest in doing more with Arduino boards. I visited many sites and read several blogs looking for inspiration for a project - I wanted to do something new, without going too crazy (this is my first stab at robotics).

Then I stumbled on an article about an IR thermometer being used with Arduino. I quickly decided I was going to do something with an IR thermometer... a heat seeking missile! Oh wait. Too dangerous.

I settled on a heat seeking vehicle. Simply put, a wheeled or tracked vehicle that would scan an area, look for areas above a certain temperature (to identify human bodies), and then drive towards that point (person).

Here are the supplies I bought in preparation:

Hitec 31055S HS-55 Economy Sub Micro Universal Servo - $11
Arduino Uno R3 - $21
2 Male DC Plug to 9v Battery Clip - $5
400 point bread board and jumper wires - $9
Double gearbox with 2 DC motors - $9
Tamiya tracked vehicle kit - $20
Motor shield for Arduino, by DF Robot - $17
MLX90614 IR Thermometer - $20

For a rough total of $112. You'll also need about $2 worth of resistors, which we'll get to shortly.

First Things First

You have to crawl before you can walk. To get started, I simply wanted to wire up the IR thermometer and do some testing. Full disclosure - I stood on the shoulders of giants throughout this project.

I followed a tutorial I found on bildr to get the basic IR thermometer wiring (PS, bildr.org is awesome, and they do a great service to the world). From a hardware pespective, the IR thermometer has 4 pins: a ground, voltage in (V-in), a "data" pin, and a "clock" pin. See the diagram below from bildr:

the V-in is the red wire, plugged into the 3.3V power supply on the Arduino.
the ground is the black wire, plugged into any of the "GND" connections on the Arduino
the blue line represents the "data" connection and should be connected to pin 4 on the Arduino
the green line represents the "clock" connection and should be connected to pin 5 on the Arduino

Courtesy of bildr.org

You'll also notice that the data and clock connections are connected to the V-in wire via 4.7 K Ohm resistors; these are called "pull up resistors". Last, the good folks at bildr included a small capacitor between the V-in and ground, but the thermometer should work without it. You'll probably want to do all of these connections on your bread board, since we're prototyping and don't want to do any soldering yet. Check out my build here:

IR Thermometer on Bread Board

Notice the bands on the resistor - yellow, purple, red, gold. This is resistor shorthand. Yellow represents 40, purple/violet is 7, red is a multiplier of 100, and gold indicates the tolerance. So these resistors are (40+7)*100 Ohms, or 4700 Ohms, or 4.7K Ohms, with a +/- 5% tolerance. I tried to replicate the colors used in the bildr example for clarity.

Alright, the IR thermometer is all wired up and connected to Arduino, the hardware portion is complete. Download all the files, including the sketch and the I2C master library from the bildr post. Don't forget to place the I2C master files in your Arduino libraries directory and restart your Arduino IDE. Now for a quick test:

Awesome! Basic temperature reading functionality established!

In Part 2, we'll incorporate the motor shield, the servo, and the chassis kit to finish the heat seeking vehicle.

AlexDGlover Blog - Articles

What's pass-reference-by-value?

Quick, naive explanation

A contradicting example

More layers to the cake

Compared to languages with pointers

Conclusion

Linkets - A Zendesk App

DNSReRoute - A Full SaaS Platform

Architecture

Third Party services

Auth0

Stripe

Database design

Static SPA Front-end

API Backend

Redirecting Backend

Conclusion

BASH Hacks as First Class Commands

Dennis Nedry Failed Login

The Move to Jekyll

Reproducing features in Jekyll

Featured Image Functionality

Site Analytics and Stats

Posting to Social Media

Retaining Permalinks

Search

Conclusion

Removing Secrets from GitHub

Process

Post Cleanup Process

DON'T PUT CREDENTIALS INTO SOURCE CONTROL

YouPassButter Slack Bot, Part 2

TL;DR Section

Where we left off

Re-structuring our commands

Dynamic Memes

OAuth and "Publishing"

Conclusion

23-Jun-2017 Update

02-Sep-2017 Update

YouPassButter Slack Bot, Part 1

TL;DR Section

Background/Context

Requirements

Mechanics

Implementation

Part 2

Issues with AWS CodeDeploy and CIS hardening

Arduino Project 7: Buzzword

Components Needed

Construction/Assembly

Arduino Project 6: Unlock it with Fire

Component List

Testing

Assembly

Share Your Salary App Part 3

Static Pages

Dynamic Pages

Deployment Notes and 'Cloudiness'

Share Your Salary App Part 2

Share Your Salary App Part 1

Project Idea/Application Premise/The Excuse

New Tools

OpenShift

MongoDB

Fat Free Framework

Swagger UI

Retrospective

The 'Database' Class

The 'Survey' Class

MongoDB and PHP Primer

Connecting

Reading a 'table'

Querying documents

Inserting a new document

Updating an existing document

Introduction to OpenShift

Initial Setup

Deployment Overview