Documentation
Performance Testing I
Practicle example: Querying the Apache log.
Model Layer Overview
IntroductionStorage of data as well as access to data in RawDev is facilitate by the data aspect or model (such as in MVC). The model is divided into three separate layers. The first layer organizes raw data into uniform entities and relations. The second layer adds security to the data by providing a mechanism to define access to data and enforcing these rules. The third layer maps entities and relationships to an object oriented interface which makes it easy to deal with nested structures and apply business logic using object oriented programming constructs.
Relational DataChances are that you primarily use a relational database such as MySql to store your data. RawDev provides uniform access to your favorite relational database as well as any other data source. Access to other data sources is handled by plugins. You can write a simple plugin that lists the files in the file system or hooks into an online data feed. After defining the structure of your data source in XML (think create table) you can access it and join it to any other data source using regular SQL. In addition, RawDev will provide mechanisms to cache data locally, copy entire data sets as well as synchronize data.SELECT b.url, b.title, p.latest_post, a.ranking FROM mysql:blog b, rss:posts p, alexa:ranking a WHERE b.url=p.url and b.domain=a.domainExample of a join across data sources listing 10 locally stored blog urls with their latest post and the alexa ranking on the blog domain. Access ControlsRawDev has powerful ways to provide data access to users . RawDev provides tools to manage users, roles, organizational groups, and provide a powerful interface to grant privileges to these entities. RawDev maps and enforces these privileges in the back-end (e.g. read/write, entity, row level, and column level access). Individual users access the data using authenticated sessions. Optionally, access is reported in a central log. It is on my wishlist to provide LDAP integration (after the summer). Object Interface The third layer maps the relational structure to an object interface. In this layer, RawDev provides additional data access tools as well as the ability for programmers to extend this. For example: say a user profile consists of many entities such as degrees, languages etc. Using this layer you have powerful tools to save the entire profile with little effort, updating only the child records that were actually changed. Further more, additional business logic can be implemented in this layer. As simple as a uniform way to display full names or as complex as the order in which records across entities can be stored given a specific business process.When do I have access to this?Although the existing production applications such as Find a Psychologist and Fodius.com enjoy significant parts of the RawDev model, the open source version is estimated to have the full model functionality in the summer of 2010. Philadelphia, Raymond Bokenkamp. |
Unit Testing
IntroductionUnit testing matters because it will make your code more reliable. This is especially true when code needs to be updated as time goes along. Unit testing takes discipline. It is tempting to just hack a result together and use ad-hoc testing. However, by recording these ad-hoc tests in actual unit tests you will be able to run them over and over again. It is not a question of whether you should do unit testing. Rather the question is: "What's the level of detail you are going to do unit testing?". Keep it basic, especially in the beginning. Unit tests are great for diagnostics and make your programming efforts more sustainable in the long run. From personal experience, I can tell you how many times I have quickly build/hacked applications together. Weeks later, when I had to change something basic, this could take hours or even a day because the change I made caused other parts to fail and it was hard to diagnose. Unit testing is a skill that can help you get to the next level as a programmer as well as an organization. It will take effort and persistence but when done in the right amount you should be able to reap benefits in weeks. RawDev has tried to make unit testing as streamlined and easy to do as possible. There are 3 aspects to unit testing in RawDev: (1) a unit test, (2) a component test (the test classes that you will write), and (3) the unit test binary that runs all, some or one unit test.Unit TestOverviewCode is organized in functions. These functions reside in objects or are static class functions. In addition, functions can interact with their environment.
Example 1A unit test typically tests either correct output of a function or verifies if the expected exception was raised. In addition one or more asserts can be done after the function was executed. Asserts are typically tests of the object and global state. Don't worry about how results of tests are displayed, I will get to that in the next section. RawDev uses fluid API calls for usability. Let's look at some examples. First a real basic example of a function that calculates the sum of 2 or more inputs. Each test has a title for diagnostics but as we will see later the title will set to the test function that implements the test.<?phpThe steps of the first test are: set function, set title, set input params, set output params, test function and evaluate results. The second function tests wheter the exception is thrown as expected. Example 2Let's look at a simple object test next.<?phpIn this example the test verifies whether the output is NULL. The assert tests wheter the "object variable" mile was set to 10. Component TestThe three components to test in RawDev are: modules (or sub-modules), classes and functions. Each of these components has a setup routine (optional), zero or more unit tests and a cleanup routine (optional). Each Component test is implemented as a test class. Creating test classes for modules and classes is often not necessary.ExampleThe example below shows all the tests for the "drive" function (see example above). Note that the unit function creates a unit test object and sets the title to the name of the function. All functions that start with test[Name] are run as unit tests. It is perfectly ok to create helper functions that do not start with "test".<?php Component Test locationTest classes are saved in the "tests" directory using a specific path/name structure:[RAWDEV_HOME]/tests/Util.php : indicates a component test for module Util. Module and Class ComponentsModule and class components are not required. In addition, they can contain setup/cleanup routines for the module or class only. Modules and classes do not have dependencies because they are expected to be tested as independent modular constructs.DependenciesA function may have implicit or explicit dependencies on other functions in the class. RawDev detects implicit dependencies by reading the source of the function. Explicit dependencies are added using the @dependencies notation in the function comment header (e.g. @dependencies __construct,start). RawDev (a) automatically rearranges the order that functions are tested based on these dependencies and (b) will unit test dependent functions if a depending function is tested in isolation.Unit Test binary[RAWDEV_HOME]/bin/runit.php is the binary that runs all unit tests it accepts zero or one argument which is the path of the component to be tested.One last note that when a component is tested, then all it's child components are tested as well. It's parent components are not tested. However the setup and cleanup routines are called for all it's parents. Test OutputThe output of any component test (or all tests) is divided into 3 parts: tree view, overall view, and failed test messages. Example:php runit.php UtilAs you can see all tests succeeded except for the "testArray" unit test function of function max in the Util class. Extra information is displayed (including a simple trace) why the test failed. Statuses: . : OK SummaryUnit Testing takes discipline. When done at the appropriate amount it will increase productivity and reliability. It is also a great diagnostic tool. RawDev organizes unit tests into component test classes. The runit binary can test one unit test, all unit tests or any component and it's children. When a function changes the state of the object or global state then multiple assert calls can be made to verify the changed state. For this reason the object or global variables should be accessible publicly or by method. Unit tests are most effective when functions have one specific task. Doing unit testing will actually alter the way you code (this is not a commercial for suits).LinksRUnitTest API Doc RComponentUnitTest API Doc |
Setup IDE + Debugger
IntroductionAlthough RawDev offers debugging
tools, it can be useful to use an IDE with a debugger. I will show you
how to setup PHP debugging using open source software for both command
line and web page debugging. This is just a quick guide to get you going, it should take you about an hour or so. This example uses eclipse as the
IDE and XDebug as the PHP module, note that this is not the only
software available. You need root access and already have apache and
php installed. The instructions are for Mac OS X / Unix but should be
very similar for Windows users. Steps* Setup XDebug * Install XDebug * Alter php.ini * Install Eclipse * Hello World example * Create Hello World example * Debug as PHP Script (command line) * Debug as PHP Web Page Setup XDebugInstall XDebug (easiest)> pecl install xdebug Install XDebug (unix / alternate)> cd /usr/local/src Alter php.iniAdd the following lines to the php.ini : # under the extensions ... Test XDebug* restart apache * create a script (info.php) in your web_root folder. * check the script and make sure XDebug is enabled see the screenshot at the bottom of document. Install Eclipse* download Eclipse for PHP Developers from: http://www.eclipse.org/downloads/download.php * install it Hello World exampleCreate example* run eclipse * >File>New>Project>PHP>PHP Project * Use default path: "~/Documents/workspace/Hello" * Project Name "Hello" [Next] [Finish] [Yes] * >File>New>PHP File * Name : index.php [Finish] Debug as PHP Script (command line)* >settings>PHP>PHP Executables>Add * Name: "php" ; Path: "/usr/local/php/bin/php" ; php.ini: "/usr/local/php.ini" ; PHP Debugger: "XDebug" [OK] * right click (in index.php source window) * select debug as PHP Script Debug as PHP Web Page* Make sure the project path is visible in the webroot * one way: create a symbolic link to project directory and project directory is visible for all (e.g. chmod 755) * test this in a browser * >settings>PHP>PHP Servers>Add (make default) * Name: "php" ; Path: "/usr/local/php/bin/php" ; php.ini: "/usr/local/php.ini" ; PHP Debugger: "XDebug" ; [OK] * right click (in index.php source window) * select debug as PHP Web Page ConclusionsObviously these are instructions just to get you started. The other debugging module is "Zend Debugger" and other PHP IDE's are available (e.g. Komodo [commercial]). Links* http://xdebug.org * http://eclipse.org phpinfo screenshot: |
Regular Expressions
IntroductionIf you use perl regular expressions on a regular basis then you probably have come across the following lines of code extensively:<?php if (preg_match($expression, $subject, $matches)) { $iCareAbout = $matches[1]; } ?>Occasionaly you also needed to replace the first matched instance. Which if you also wanted to check if matches were replaced and what the matches are is more cumbersome: <?php if (preg_match($expression, $subject, $matches)) { $iCareAbout = $matches[1]; preg_replace($expression, $replace, $subject, 1); } ?>For both of these instances the RawDev preg function looks as follows: <?php $iCareAbout = RUtil::preg($expression, $subject); # case 1 $iCareAbout = RUtil::preg($expression, $subject, $replace); # case 2 ?>In addition, the RawDev preg function can match/replace all instances. Lastly, it can handle an array of strings as the subject of which only the strings that matched were returned. APIstatic mixed function preg($regex, $subject, $replace, $all, $alwaysArray)
ExampleThis example works on the string: "[USA] Gold, [Netherlands] Silver, [Canada] Bronze" It matches/replaces the first country and then all countries. The result is:<?php require_once('rawdev/RawDev.php'); require_once(RAWDEV_LIB.'/Util/Util.php'); $str = "[USA] Gold, [Netherlands] Silver, [Canada] Bronze"; $str1 = $str; $str2 = $str; $str3 = $str; $str4 = $str; # find first country $result1 = RUtil::preg("/\[(.*?)\]/", $str1); # find and replace first country $result2 = RUtil::preg("/\[(.*?)\]/", $str2, "\\1"); # find all countries $result3 = RUtil::preg("/\[(.*?)\]/", $str3, NULL, true); # find and replace all countries $result4 = RUtil::preg("/\[(.*?)\]/", $str4, "\\1", true); ?> ConclusionUsing RUtil::preg code will reduce the amount of code you produce. Which will, in the long run, save (a) programming hours, (b) reduce bugs and (c) increase execution performance (less code to put in memory). The subject is changed by reference. This can be annoying because you need to assign a temporary var if you don't want to change the subject. Always use preg_replace when you don't care about returning matches.LinksRUtil API Doc |
Debugging: Variable Browsing
IntroductionUsing this library you can browse the content of complex variables, one nested level at the time by using commands such as zoomin, zoomout and exit. Currently, this class should only be used for command line applications. Web browsable functionality will be added later using through firephp/firebug.ExampleBy executing the snippet below you will see the following screen. At this point you can decide to exit (or zoomout), or zoom in to nested variable links [1] and [2].<?php require_once("rawdev/RawDev.php"); require_once(RAWDEV_LIB.'/Util/Dumper.php'); class Test { var $a='hi'; var $b = array(1, 2); } $complex = array(1, 'hi', array(99, 17), 'std' => new stdClass(), new Test(), array()); RDumper::dump($complex); ?> ConclusionFor command line application development this beats var_dump and will save you a ton of time when debugging.LinksRDumper API DocFuture enhancementsColor coding? |
Stdin
IntroductionRStdin is tiny class that deals with interactive keyboard input from the command line. For testing purposes the keyboard input can be faked.exampleThis example first shows regular use. The output is trimmed by default but this can be changed (see RStdin API Doc). In the second part you can see how the keyboard input can be faked for testing purposes.<?php require_once("rawdev/RawDev.php"); require_once(RAWDEV_LIB.'/Util/Stdin.php'); # regular use $input = RStdin::read('Please enter input: '); print "$input"; # testing use RStdin::$buffer[] = 'hello'; # for testing purposes the input buffer can be set $input = RStdin::read('Please enter input: '); print "$input"; ?> ConclusionThis class is useful when dealing with command line applications. RawDev uses this class in the Dumper utility in which you can interactively view nested complex data structures.LinksRStdin API Doc |
Error Reporting
IntroductionIn RawDev, all PHP errors and exceptions can be handled centrally using the Error Reporting library. In addition, RawDev gives more options of error messages than the standard PHP error_reporting level. For example, by default strict errors such as illegal variable references are returned but strict errors such as undefined array offsets are not. RawDev finds that being more strict with variable (and object property) references leads to catching programming mistakes earlier in the development process. Most RawDev users will find this class extremely useful. ExampleIn the example below, I trigger several PHP errors: invalid offset, illegal variable reference and illegal object property reference. As you can see, the invalid offset is ignored by default and both other errors are thrown as RExceptions (RawDev Exceptions). Any other RExceptions can be handled by a central handler as well. <?php require_once('rawdev/RawDev.php'); require_once(RAWDEV_LIB.'/Core/ErrorReporting.php'); RErrorReporting::singleton()->initialize(); $b = array(); if ($b[0]); # this will not trigger an error $difficultVariable = 17; try { if ($difficultVaariable); # this triggers an error because the wrong variable is used. } catch (RException $e) { print $e->getMessage()."\n"; } class Test { } $a = new Test(); try { if ($a->b); # this triggers an error because an invalid object variable is used. } catch (RException $e) { print $e->getMessage()."\n"; } ?>; ConclusionThis is just one of these classes that as a developer you will get attached to because it forces you to code a little tighter but avoids having to debug illegal variable and object property mistakes. The central error handling is a nice bonus. Links |
What is RawDev?
Imagine a scenario: I have survey results in a Google Doc that gives me a list of websites that my users like. I have a website like http://www.alexa.com/ that provides rankings of websites worldwide. I have a table of data on websites in Oracle. Three data sources, three disparate data sets. But looking at it, we know that there are linkages there. Imagine an application framework that allowed you to query all three, simultaneously, and seamlessly present the data to your users in a convenient front-end. Now imagine that framework is called RawDev. This sort of project is exactly what RawDev is designed to do: bring together independent data and give it to an end-user in a seamless presentation. You set up the datastore once, and then build apps to use that data again and again. Integrate different types, different locations, even different storage engines of data and have one interface. View all these different data-sets as one data-set in your favorite database gui application. Oh, and RawDev will throw in powerful XML based access controls as well. You can download a beta version of RawDev this summer after a life demo at the UI Conference at UPENN on July 21st and July 22nd. In the mean time you can get early bird access and follow the development process on this blog and http://rawdev.bokenkamp.com. |