Web Appers: One of my go to sites for web development resources

Weappers screenshot

Web Appers is one of the sites I visit regularly to learn about new web development tools and resources.

The website says:

WebAppers is a blog dedicated to share top quality open source resources for web developer and web designer daily. As a web designer, you’ll find
some of the best free icons, stock photos, brushes, fonts and design inspirations. As a web developer, you’ll also find some of the best Javascript
and Ajax components like modal windows, menus, galleries, tooltips, charts, calendars plugins and a lot more …

Email to my wife’s cousin: Software development resources

One of my wife’s cousins is graduating from college in the spring, and she was offered a job doing software development. She called and we spoke about what it’s like to be a software developer day-to-day. When we were done talking she asked me if I had any resources I could share to help her get started with learning about Java development. Here’s what I sent her.


Hey Cuz!

Below are three books that might be of interest to you. I don’t expect you to read all three. Most developers are lucky to read one book a year. These are just suggestions of books you might want to know about. I’d start with Head First Java.

Also, I have listed several concepts you might want to be familiar with. DON’T FREAK OUT! I know it’s a lot of information. I don’t expect you to learn all of that stuff. Just skim it so that if someone says, “we deploy our application on Tomcat,” you will have seen the word “Tomcat” before.

I recommend that you download a copy of the Eclipse IDE, install it, and use it to work through some examples from Head First Java.

Feel free to reach out to me if you have any more questions. You can reach me at this email address or on my cell phone at [REDACTED SON!].

Good luck!

Books

Head First Java, 2nd Edition by Kathy Sierra

If you want to learn basic Java programming, this is where I would start.

Head First Java Cover


Effective Java (2nd Edition) by Joshua Bloch

This is my favorite Java book. I refer to it regularly. It’s the only one I have read cover to cover.

Effective Java Cover


Code Complete: A Practical Handbook of Software Construction, 2nd Edition by Steve McConnell

This book is more about programming in general. Pick up a copy from your library if you can and thumb through it. It might have some interesting stuff in it.

Code Complete Cover

Useful Website

Stack Overflow: http://stackoverflow.com/

This is the largest question and answer site for programmers. It is an invaluable resource if you have programming questions. Here’s a link to my Stack Overflow profile page if you want to see the kinds of questions a web developer might ask.

Concepts

Again, DON’T FREAK OUT! This is just a list of stuff an enterprise Java developer might work with. Just skim some of the web pages and familiarize yourself with some of the concepts if you have time.

StackOverflow: An invaluable tool in my developer toolbelt

stack-overflow-logo.png

It feels odd to even mention this site since I use it so often, but I still meet software developers who have never heard of Stack Overflow.

Stack Overflow is a question and answer site for professional and enthusiast programmers.” I’ve asked and answered a number of questions on the site, and it’s my go to place on the web when I have a software development or programming question of any kind. When I see Stack Overflow in my search results, it’s usually the link I click on first.

Here’s a link to my profile if you want to see the kinds of questions you can ask and get answers to.

gevent: coroutine-based Python networking library

I learned about this from the FLOSS Weekly podcast . I’ve become more interested in event-driven frameworks lately because of Node.js and Vert.x.

From the gevent website:

gevent is a coroutine-based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev event loop.

Features include:

  • Fast event loop based on libev (epoll on Linux, kqueue on FreeBSD).
  • Lightweight execution units based on greenlet.
  • API that re-uses concepts from the Python standard library (for example there are Events and Queues).
  • Cooperative sockets with SSL support »
  • DNS queries performed through threadpool or c-ares.
  • Monkey patching utility to get 3rd party modules to become cooperative

GoAccess: Pretty Web log parser that runs in the terminal

GoAccess dashboard screenshot

From the GoAccess website:

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems. It provides fast and valuable
HTTP statistics for system administrators that require a visual server report on the fly.

Features

  • General Statistics, bandwidth, etc.
  • Time taken to serve the request (useful to track pages that are slowing down your site)
  • Top Visitors
  • Requested files
  • Requested static files, images, swf, js, etc.
  • 404 or Not Found
  • Hosts, Reverse DNS, IP Location
  • Operating Systems
  • Browsers and Spiders
  • Referring Sites
  • Referrers URLs
  • Keyphrases
  • Geo Location – Continents/Countries New
  • HTTP Status Codes
  • Ability to output JSON and CSV New
  • Different Color Schemes
  • Support for IPv6
  • Unlimited log file size
  • Output statistics to HTML. See report. N

Supported Formats

  • Any custom log format string
  • Common Log Format (CLF) Apache
  • Combined Format (XLF/ELF) Apache|Nginx
  • W3C format (IIS)
  • Amazon CloudFront (Download Distribution).
  • Apache virtual hosts

Output Formats

  • Terminal (default)
  • HTML
  • JSON
  • CSV

xip.io

xip.io/

This would have come in handy when I was doing a lot of testing of our mobile
apps. At work we have a “live” silo and a “dark” silo. For large deployments,
we stage the new builds in the “dark” silo and test it there. Then when it’s
time to go live with the new builds, we update the load balancers to swap the
silos and make the new code live. Pointing native apps to the dark silo
is kinda tricky since there is no /etc/hosts/ file to edit. I have been
testing by running my own DNS server on my laptop, but that is a pain. I’m
going to play with xip.io to see if it can help me with this kind of
testing.

From the website:

xip.io is a magic domain name that provides wildcard DNS
for any IP address. Say your LAN IP address is 10.0.0.1.
Using xip.io,

          10.0.0.1.xip.io   resolves to   10.0.0.1
      www.10.0.0.1.xip.io   resolves to   10.0.0.1
   mysite.10.0.0.1.xip.io   resolves to   10.0.0.1
  foo.bar.10.0.0.1.xip.io   resolves to   10.0.0.1

…and so on. You can use these domains to access virtual
hosts on your development web server from devices on your
local network, like iPads, iPhones, and other computers.
No configuration required!

GitList

GitList

Another team at work has begun using GitList as their Git repository
viewer.

It looks like a nice self-hosted alternative to GitHub

From the GitList website:

GitList allows you to browse repositories using your favorite browser,
viewing files under different revisions, commit history and diffs. GitList is
free and open source software, written in PHP, on top of Silex and the Twig
template engine.

Scraping Data from a Website using JSoup (Lightly Seasoned with SQLite)

Here’s some Java code I wrote to scrape links from a site for insertion into a
SQLite database. I use JSoup to do the scraping. It uses JQuery selectors
to select content to scrape. I think I prefer XPath, but it’s good to
have another tool in the toolbox. I use a Google Chrome extension called
Scraper to help me form the JQuery selectors. In this example, I’m using
the SQLite FTS4 Fulltext Search extension to make the values easily
searchable in the DB.

package com.sample;

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

/**
 *
 * @author Oscar Grouch
 */
public class SampleScraper {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) 
    {
        /* Pull the all posts page */
        Document doc = null;
        try 
        {
            doc = Jsoup.connect("http://localhost/all.html").get();
        } 
        catch (IOException ex) 
        {
            Logger.getLogger(Scraper.class.getName()).log(Level.SEVERE, null, ex);
        }

        /* Get a list of all of the links on the all posts page */
        Elements links = doc.select("h2 > a");
        if (links != null)
        {
            /* Initialize DB driver. */
            try 
            {
                /* load the sqlite-JDBC driver using the current class loader */
                Class.forName("org.sqlite.JDBC");
            } 
            catch (ClassNotFoundException ex) 
            {
                System.err.println("Class not found: org.sqlite.JDBC: " + ex.getMessage());
            }

            Connection connection;
            try
            {
                /* create a database connection */
                connection = DriverManager.getConnection("jdbc:sqlite:C:/Temp/sample.db");
                Statement statement = connection.createStatement();
                statement.setQueryTimeout(30);  // set timeout to 30 sec.
                statement.executeUpdate("drop table if exists content");
                statement.executeUpdate("CREATE VIRTUAL TABLE content " + 
                                        "USING fts4(title, url, post)");
            }
            catch(SQLException e)
            {
                // if the error message is "out of memory", 
                // it probably means no database file is found
                System.err.println(e.getMessage());
                return;
            }


            /* Loop through the links */
            for (Element link : links)
            {
                /* Print out the link text */
                System.out.println("Processing " + link.text() + "...");

                Document currentDoc = null;
                try 
                {
                    /* Pull the current link page */
                    currentDoc = Jsoup.connect(link.attr("abs:href")).get();
                } 
                catch (IOException ex) 
                {
                    Logger.getLogger(Scraper.class.getName()).log(Level.SEVERE, null, ex);
                }

                /* Get the post Text */
                Element post = currentDoc.select("div.post").get(0);
                String postText = post.text();

                /* Print the post text */
                System.out.println("Post Text: " + postText);

                try
                {
                    PreparedStatement preparedStatement 
                      = connection.prepareStatement("insert into content values(?, ?, ? )");
                    preparedStatement.setString(1, link.text());
                    preparedStatement.setString(2, link.attr("abs:href"));
                    preparedStatement.setString(3, postText);
                    boolean isResultSet = preparedStatement.execute();
                    if (isResultSet)
                    {
                        System.out.println("A ResultSet was returned.");
                    }
                    else
                    {
                        System.out.println("No ResultSet was returned.");
                    }
                }
                catch(SQLException e)
                {
                    // if the error message is "out of memory", 
                    // it probably means no database file is found
                    System.err.println(e.getMessage());
                }

            }
        }
    }
}