Timewarp Technologies, LLC - Hints and Tips

Bring rogue programmers in from the cold.

jlopez — Tue, 16 May 2006 15:58:57 +0000

One interesting resource for programmers is "The Daily WTF". It is a great place to read about things that should *not* be happening in your organization: antipatterns for code and business processes (mostly code).

A recent "The Daily WTF" discusses the plight of a company where a “rogue” programmer built an inefficient spreadsheet for a supervisor, only to find it was too popular and those inefficiencies brought down the database by overloading it with connections. Operating outside of IT, the understanding of enterprise requirements was not there. On the other hand, many have argued that rogue programmers exist because of unresponsive IT departments, and they provide a valuable service.

The irony is that “rogue” programmers can easily be replaced in most companies by a small number of junior programmers that report to IT itself. Once a skunk works application is conceived and built, IT can use it as a prototype for a real solution to the problem.

In the article, the rogue programmers “application” is a nightmare for IT: because they didn't review it and bring it up to standards. On the other hand, any IT department that can't take a working (even if for a single user) application written by an outsider and convert it into a real, audited application in short order needs to be sacked. The specification is a functional prototype made using tools that limit expressiveness and probably with a skill set that is simplistic at best. If IT can’t trump that quickly and simply, it is time to reconsider your IT department.

Our solution to skunk works projects: we encourage them, but we insist that once they are functional as prototypes that IT rework them to fit into the infrastructure. We maintain several real time replicas of the databases; skunk works projects operate off those to avoid disruption of production.

It turns out your users probably know their requirements better that the IT gurus... and can implement a solution or partial solution. On the other hand, only IT can assure integration problems won't ensure. Use each for their strengths and your library of applications will grow with the combination of user knowledge *and* IT robustness. It is ironic that Agile Methods are all the rage; yet realizing that the initial prototype phase can be very effective in the hands of the central stakeholder seems verboten.

Category:

Hints and Tips

Excel Spreadsheet Transformations

jlopez — Wed, 10 May 2006 20:34:39 +0000

Today I had an request to transform data that came from a printed report into usable Excel data. The data had been printed to PDF, so a PDF to Excel converter tool did the bulk of the work. However, even after being placed in Excel, there were two remaining requirements to achieve this goal. The first was to remove the recurring headers and footers of the report. The second was to reassemble each reported item's three lines of data back into to a single coherent row of data.

To do the initial transformation (removing the headers) I used the following VB for Applications code:

Sub RemoveHeaders()
Dim CellRow As Integer, CellCol As Integer

CellRow = 1
CellCol = 1
Dim eof As Integer
eof = 0
Do Until eof = 1
    Select Case Cells(CellRow, CellCol).Value
    Case "EOF" 'We have found the manually inserted EOF. Stop.
        Exit Do
    Case "MIGBCL01" 'We have found the page header tag. 
        'The next two lines are not actually used in the final run, 
        'but were mearly a visual confirmation that the targeted lines were accurate.
        'Set myRange = range(Cells(CellRow, CellCol), Cells(CellRow + 13, CellCol))
        'myRange.Select

        'Remove the row and the 13 rows after it.
        For i = CellRow To CellRow + 13
            rows(CellRow).Delete
        Next
    Case Else 'We have an ordinary row, continue
        CellRow = CellRow + 1
    End Select
Loop
Cells(CellRow, CellCol).Select 'Select EOF, we are done.

End Sub

This is a very simple routine, but hopefully others will find the ideas in it helpful when confronted with data that is oddly formatted. There were a few manually performed steps: first was the insertion of the EOF to stop the routine and the second was manually handling the totals at the bottom of the report. However, those changes only took seconds to perform: the bulk of the work was removing the hundreds of page breaks. The code completed in under a minute (and would have been faster if Application.ScreenUpdating had been set to false for the actual run).

After getting the headers out of the way, it was time to transform the data itself. First, I created extra columns to move the data into:

Sub createNewColumns()
'This routine creates two extra columns for each existing
'column. This will allow dumping the data from the rows
'below a main row into the column.
Dim i As Integer
For i = 11 To 2 Step -1 'We have 11 columns,
    'we wish to insert two more for each to "flatten" the data.
    Columns(i).Insert
    Columns(i).Insert
Next
End Sub

Then we needed to move the data itself. Here we loop over the range, migrating the data from the old rows into the new columns.

Sub moveData()
'Move through the data grid, tranferring data from columns
'into a single row.

Dim startRow As Integer 'The first row of a data group.
Dim columnNumber As Integer 'The column to work in
Dim dataRow As Integer 'The subdata row to work with
For startRow = 13 To 3249 Step 4 'Hard coded start, end and skip length for run.
    For columnNumber = 1 To 31 Step 3 'Gap is 4, only 3 rows of data, one blank
        For dataRow = 1 To 2 'Simply 1,2 to get at the rows below the main.
            Cells(startRow, columnNumber + dataRow) = _
                Cells(startRow + dataRow, columnNumber) 'Move from rows to columns
        Next dataRow
    Next columnNumber
Next startRow
End Sub

Finally we remove the extra data rows, which are no longer necessary with the data in the new columns

Sub removeExtraRows()
'Remove the extra rows now that data is in single rows.
Dim CellRow As Integer, i As Integer
CellRow = 13 'Start on first data line
Do
    If Cells(CellRow, 1) = "EOF" Then
        Exit Do 'We are done.
    End If
    
    CellRow = CellRow + 1
    For i = 1 To 3 'three extra rows each.
        rows(CellRow).Delete
    Next
Loop
Cells(CellRow, 1).Select 'We are done, select EOF
End Sub

With these steps performed, I formatted a few columns to improve data display and forwarded the transformed spreadsheet on to the end user. I have not turned this into a generic tool because each situation will be different and require a lot of cusomization, but hopefully these code snippets will help someone looking for a template on how to manipluate Excel spreadsheets.

Category:

Hints and Tips

Domain specific languages, are they worth it?

jlopez — Fri, 21 Apr 2006 18:13:07 +0000

Yet another interesting discussion of Domain Specific Languages popped up:

http://www.artima.com/forums/flat.jsp?forum=106&thread=155752

Domain specific languages (DSLs) are scripting languages (usually) built on top of a traditional environment. For programmers who have written code only in a single language model, moving to a DSL model can be a bit of a shock. The fundamental idea is that when your base programming language is becoming constraining in some way, you can build a DSL on top of it to remove some of the limitations. (Another option is migrate to a more powerful language, but sometimes that is too drastic of a step to take.)

A project I worked heavily on uses a code generator. Code generators are actually very similar to domain specific languages in the problem set that they solve. In this specific case, the code was being written in VBScript for use in an Active Server Page application. The application created long before .NET 1.0 was released. Over time, it became clear that the same fundamental ideas were being used over and over in some parts of our system.

Instead of repeating ourselves, we built a data driven code generator. We provided non programmers with a UI to edit the system components without actually writing code. The data in the database created by the non programmers was in effect a DSL that was specified the generated code. Our DSL allowed us to create the structure of the required code, with the generated code being fleshed out on demand to the specifics based on the data provided by our non programming staff. The final product allowed us to reintroduced inheritance into VBScript via our DSL, reducing our actual source code size while increasing productivity.

This kind of solution works well in some cases, but can also create debugging problems. Your crashes will probably only point to generated code, which you must then map back to the original source of the problem. On the other hand, over time the DSL can become very robust as edge cases are dealt with and debugging routines are implemented.

We are currently porting the application from VBScript to C#. We are seeing great improvements simply by building interpreters of the original DSL. The old system would generate code which then was saved to ASP pages on disk which were invoked by end users. The C# based interpreters are seeing response times improve because many of the ugly VBScript hacks have given way to a more elegant C# based class structure and a more direct implementation of our ideas.

All of this brings up the question: are DSLs worth the effort and potential liabilities? One limitation is that DSLs are not usually strongly typed while languages such as C# and Java are. This means you can create more errors that are found late in the process with DSLs. Another limitation is that your core language has robust debugging tools and a large library to call upon. On the other hand, C# and Java can be very cumbersome to use if you need to invoke many variants of nearly identical code. You could subclass for each instance, but in many cases DSL driven invocations of a more flexible object are more efficient than generating reams of nearly identical code. As long as the work to build and maintain the DSL is lower than subclassing, it is a winning choice.

After many years of using these types of solutions, I am finding that the best way to use DSLs is as a guide for what your core system really needs to be able to do. The dividing line between DSL and core code will evolve as the system grows. Early on, you will probably have all functionality in the core system. As your requirements demand more flexibility, you will find yourself repeating yourself. Where possible, use the core system’s language to eliminate that repetition: you will retain the advantages of strong typing and a robust debugging environment. In many cases you can avoid creating a DSL and the work maintain it by clever analysis of the core problem.

However, if you find that you are repeating yourself in ways that your core language cannot easily remove, consider moving to a DSL or data driven architecture. Over time, this flexibility will lead to the discovery of features that probably should be submerged back into the core system. You will find these wherever you find yourself repeating yourself in the DSL (which defeats the purpose of the DSL) or creating large blocks of code in the DSL. When these things happen, it indicates that your core system needs to gain more power, potentially as an enhancement to the DSL you created. Remember, the goal is to write as little code as possible in the DSL side: verbose DSL code is an anti-pattern to be avoided and too much dependence on a DSL can create a version of the second system effect where too much work is being put into the DSL and not enough on solving the original problem.

Category:

Hints and Tips

Google Analytics

jlopez — Tue, 22 Nov 2005 21:16:32 +0000

Google is testing a new service: "Google Analytics". If you already use Ad Words, then this additional reporting can be useful for determining which words are getting you the best results. More than a hit counter, the product allows you to monitor four items to help you understand your marketing successes and failures.

Campaign tracking variables: These are URL elements you can use in web advertisements, e-mails, etc. These codes allow you to track the number of users arriving from different marketing efforts. Most marketing people have figured out how to use different landing pages or similar variables to track marketing responses, but these are particularly useful when combined with the other Analytics features.
Goals: Pages that you want the visitor to reach. For an e-commerce site, this would be the confirmation after a successful purchase. For a blog, it might be the "signup for an account" page.
Funnels: A chain of pages that a user would traverse to reach a goal. This will allow you to see the "drop off" rate at different parts of the process. How many users balk at using the checkout confirmation screen, the shipping screen, the credit card page, etc. By understanding where you lose people, you can make adjustments to improve that step.
Filters: Allows for the exclusion of various elements from the reports (such as your companies IP range, to avoid inflation from internal user views).

This is information that can be gleaned from traditional tools, but the slick implementation really shines due to the goals and funnels implementation. These two tools give a simpler and yet more detailed explanation of user flow than I have seen before, and it is free.

Category:

Hints and Tips

FireDaemon

jlopez — Tue, 11 Oct 2005 22:55:10 +0000

From time to time it is nice to be able to run a program as a service. If you have written the software yourself (especially with .NET) creating services isn't actually all that hard. However, for third party programs or as a quick fix, nothing beats FireDaemon. FireDaemon is a low cost solution that can convert nearly any executable into a daemon. Simply specify the executable to run in the background and set the startup properties, security account, command line, etc and you are off. This is ''particularly'' useful for those programs that seem like they should have been background capable to start with, such as some backup tools that only operate while a user is logged in. Yes, these are usually inexpensive options and do what they were created to do quite well, but the ability to background them makes them even more useful. Iomega's backup tools or Second Copy 2000 both benefit from the ability to run backgrounded, and after doing so you wonder why they didn't in the first place.

Category:

Hints and Tips

LLBLGen

jlopez — Fri, 07 Oct 2005 17:35:44 +0000

If you are coding .NET projects, one of features I miss most compared to Java is a good Object Relational Mapping tool. Java programmers for years have been using tools such as Hibernate to help with persistence code, but for .NET the tools have been fairly lacking. Although there are products such as NHibernate, they are still not as mature as the Java versions. Commercial offerings are usually very expensive. Recently I tried LLBLGen and found it to be a great solution with good support and a good feature set. On top of that, the price makes it a no-brainer option for even an independent developer working on a budget.

When installing the product, copy your license file into the program’s main directory (by default “C:\Program Files\Solutions Design\LLBLGen Pro”). When you generate code, you will need to reference your generated projects ( and DBSpecific) and the SD.LLBLGen.Pro.ORMSupportClasses.NET library. Finally, in the DBSpecific Project, drag the DatabaseSpecific/App.Config to your application’s main project (or add the contents of that file if you already have an app.config). With all of these steps done, you can access all the features of the product.

Make sure you read the section on “concepts” as it makes clear the underlying theory behind the product, which is good to understand. I wish all documentation was this clear. I will extend this story as I dig deeper in to this wonderful product.

A couple of changes to my normal pattern of table naming was required: normally I distinguish tables and view with tblName and viewName style prefixes. This product strips prefixes, so the preferred solution is to give table names singular names, and view plural names. This becomes important because to "firehose" data, you will want to use the TypedView. As an example, I have PolicyStatus and PolicyStatuses as the table and view respectively.

One thing to watch for: if you save an entity and wish to continue using it, you must reload the entity after the save or the system will complain that your copy is "out of synchronization".

this.adapter.SaveEntity(m_data);
this.adapter.FetchEntity(m_data); //Reload after saveA

Combo box value loads (and other firehose cursors) are best implemented by using a stored procedure and then loading from the result set (sorting provided by the procedure):

DataTable surplusLinesStatuses = RetrievalProcedures.RpSurplusLinesStatuses();
this.PolicySurplusLinesStatus.LoadTableItems(surplusLinesStatuses, "Status", "SurplusLinesStatusID");

Note that the order in which the combo box properties are set is important (this is found in the LoadTableItems method of my derived combo box object type):

this.BeginUpdate();
this.ValueMember = valueMember;
this.DisplayMember = displayMember; 
this.DataSource = data; //Set the members *before* setting the source. 
this.EndUpdate();

Note that if the data source was set *first* this would crash. This appears to be a .NET issue as it makes no sense to worry about order in which these are assigned. It only crashes if one of the items (usually the value member) ends up with a name containing ID at the end.

Pulling a collection of data in is quite easy:

CompanyEntityFactory companyEntityFactory = new CompanyEntityFactory();
EntityCollection companies = new EntityCollection(companyEntityFactory); 
ISortExpression sorter = new SortExpression(
  SortClauseFactory.Create(CompanyFieldIndex.CompanyName, SortOperator.Ascending));
adapter.FetchEntityCollection(companies, null, 0, sorter);

The most complex part of this is the sort expression. We use factories to generate the objects that are relevant, and this takes some getting use to. Also, naming conventions are critical, each table is TablenameEntity and has TablenameFieldIndex and TablenameEntityFactor objects created. The use is illustrated above.

A more complex example, here using another object to collect up the rows based on a relationship and with three sort filters:

CompanyEntity companyEntity = new CompanyEntity(CompanyGUID);
adapter.FetchEntity(companyEntity);

ISortExpression sorter = new SortExpression(
  SortClauseFactory.Create(LocationFieldIndex.State, SortOperator.Ascending));
sorter.Add(
  SortClauseFactory.Create(LocationFieldIndex.City,  SortOperator.Ascending));
sorter.Add(
  SortClauseFactory.Create(LocationFieldIndex.Zip, SortOperator.Ascending));

EntityCollection locations = companyEntity.Location;
adapter.FetchEntityCollection(locations, companyEntity.GetRelationInfoLocation(), 0, sorter);

Note the companyEntity.GetRelationInfoLocation() that is passed: this is how we restrict the collection to *related* records. Passing null here pulls everything in the table (which is what you want sometimes, but not if you went to the trouble of pulling in a related record most times.

Another action you may wish to perform is a direct update of data without loading the entity. Here is an example, updating a field without loading it (this is the equivelent of an UPDATE Login SET PasswordHash = @NewPassword WHERE LoginGuid = @LoginGuid followed by the parameter sets, connection creation and execution).

LoginEntity loginEntity = new LoginEntity();
loginEntity.LoginGuid = m_loginEntity.LoginGuid; //Update the current user
loginEntity.IsNew = false; //We exist in the DB already.
loginEntity.passhwordHash = this.newPassword.Text;
DataAccessAdapter adapter = new DataAccessAdapter();
adapter.SaveEntity(loginEntity);

When using "self servicing" mode files, this next point is moot. When using adapter to control data access more carefully (which is how I would recommend the product be used, despite the convenience of self servicing mode) letting the adapter know what related objects to bring along with is important to the performance of the program. In this example, we are pulling all of the policies that relate to an insured:

InsuredEntity insuredEntity = new InsuredEntity(InsuredGUID);
adapter.FetchEntity(insuredEntity);

SortExpression sorter = new SortExpression(
    SortClauseFactory.Create(PolicyProspectFieldIndex.EffectiveDate, SortOperator.Ascending));

EntityCollection policies = insuredEntity.PolicyProspect;
PrefetchPath2 prefetchPath = new PrefetchPath2((int)EntityType.PolicyProspectEntity);
prefetchPath.Add(PolicyProspectEntity.PrefetchPathPolicyType);
adapter.FetchEntityCollection(policies, insuredEntity.GetRelationInfoPolicyProspect(), 0, sorter, prefetchPath);

foreach(PolicyProspectEntity policyProspectEntity in policies)
{
...
}

He we see an entity relationship being used (the insured's policies can be found by filtering on insuredEntity.GetRelationInfoPolicyProspect(). We are sorting these entries by a date field, again similar to prior examples. The novel piece of this code is the use of a prefetchPath, here constructed with the PolicyProspectEntity and then given the PolicyProspectEntity.PrefetchPolicyType as an related entity to load. Running the SQL Profiler while this runs shows the following queries executed:

SELECT 
FROM Insured  
WHERE ( Insured.InsuredGUID = @InsuredGuid1)

SELECT 
FROM PolicyProspect  
WHERE ( PolicyProspect.InsuredGUID = @InsuredGuid1) 
ORDER BY PolicyProspect.EffectiveDate ASC

SELECT 
FROM PolicyType  
WHERE ( PolicyType.PolicyTypeID IN (
	SELECT PolicyProspect.PolicyTypeID AS PolicyTypeId 
	FROM PolicyProspect  
	WHERE ( PolicyProspect.InsuredGUID = @InsuredGuid1)))

Note that this is potentially more efficient than the standard join query, where the same policy type information might be repeated over and over again in the result set. Here we see that only those policy types that are relevant are preloaded, and each will only be preloaded only once. Compare that to the equivalent SQL code:

SELECT 
FROM Insured  
WHERE ( Insured.InsuredGUID = @InsuredGuid1)

select *
from PolicyProspect 
inner join PolicyType 
  on PolicyProspect.PolicyTypeID = PolicyType.PolicyTypeID
where PolicyProspect.InsuredGUID = @InsuredGUID

The direct SQL only makes two round trips instead of three, but potentially repeats policy type information needlessly.

Some typical patterns:

using System;
using ORM;
using ORM.DatabaseSpecific;
using ORM.EntityClasses;
using ORM.FactoryClasses;
using ORM.HelperClasses;
using SD.LLBLGen.Pro.ORMSupportClasses;

namespace PolicyTracker.BL
{
    internal static class AgentManager
    {
        internal static AgentEntity CreateAgent(Guid locationGuid)
        {
            AgentEntity entity = new AgentEntity();
            entity.AgentGuid = Guid.NewGuid();
            entity.LocationGuid = locationGuid;
            return entity;
        }
        
        internal static string GetCaption(AgentEntity agent)
        {
            return agent.AgentName;
        }

        internal static AgentEntity GetAgent(Guid agentGuid)
        {
            AgentEntity agent = new AgentEntity(agentGuid);

            using (DataAccessAdapter adapter = new DataAccessAdapter())
                adapter.FetchEntity(agent);

            return agent;
        }

        internal static EntityCollection GetReassignTargets(AgentEntity sourceAgent)
        {
            LocationEntity sourceLocation = new LocationEntity(sourceAgent.LocationGuid); //Agent to load
            PrefetchPath2 path = new PrefetchPath2((int) EntityType.LocationEntity); //Get the location of the agent.
            path.Add(LocationEntity.PrefetchPathCompany).SubPath.Add(CompanyEntity.PrefetchPathLocation);

            using (DataAccessAdapter adapter = new DataAccessAdapter())
            {
                adapter.FetchEntity(sourceLocation, path);
            }

            return sourceLocation.Company.Location;
        }

        internal static void Reassign(AgentEntity sourceAgent, LocationEntity targetLocation)
        {
            sourceAgent.LocationGuid = targetLocation.LocationGuid;

            using (DataAccessAdapter adapter = new DataAccessAdapter())
            {
                adapter.SaveEntity(sourceAgent);
            }
        }

        internal static EntityCollection GetInsureds(Guid agentGuid)
        {
            AgentEntity agentEntity = new AgentEntity(agentGuid);
            DataAccessAdapter adapter = new DataAccessAdapter();
            adapter.FetchEntity(agentEntity);

            SortExpression sorter = new SortExpression(
                SortClauseFactory.Create(InsuredFieldIndex.Name, SortOperator.Ascending));

            EntityCollection insureds = agentEntity.Insured;
            adapter.FetchEntityCollection(insureds, agentEntity.GetRelationInfoInsured(), 0, sorter);

            return insureds;
        }

        internal static LocationEntity GetLocation(AgentEntity agent)
        {
            return LocationManager.GetLocation(agent.LocationGuid);            
        }

        internal static InsuredEntity CreateChildInsured(AgentEntity agent)
        {
            InsuredEntity newInsured = new InsuredEntity();
            newInsured.InsuredGuid = Guid.NewGuid();
            newInsured.AgentGuid = agent.AgentGuid;
            return newInsured;
        }
    }
}

An example of an update performed against the data store (no loading of objects):

        /// <summary>
        /// Moves the child Locations to the new parent Company
        /// 
        /// update Location 
        /// set CompanyGuid = @targetGuid
        /// where CompanyGuid = @sourceGuid
        /// </summary>
        /// <param name="sourceGuid">Company to remove children from.</param>
        /// <param name="targetGuid">Company to add children to.</param>
        internal static void MergeChildEntities(Guid sourceGuid, Guid targetGuid)
        {
            LocationEntity updateLocation = new LocationEntity();
            updateLocation.CompanyGuid = targetGuid;

            RelationPredicateBucket filter = 
                new RelationPredicateBucket(LocationFields.CompanyGuid == sourceGuid);
            
            using (DataAccessAdapter adapter = new DataAccessAdapter())
            {
                adapter.UpdateEntitiesDirectly(updateLocation, filter);
            }            
        }

A note about migration between environments.

LLBLGen writes code that includes an explicit reference to the database being referenced. This is because the ORM can talk to multiple databases at once. However, this also introduces the requirement to be able to rewrite the database references on the fly: if your production or test environments have databases with different names, the config files must be updated to rewrite the "source" names to the new names.

Adding

to the .config file for the application will ensure the proper code is executed against the database. Simply changing your connection string will *not* have the desired effect unless the exact same names are used for the databases. There is also a schema rewriting section that works the same, just at the schema level.

Update: an example of a sub-query. Here we are looking for the current record. If the effective date is in the future or is an older record, it is excluded.

FieldCompareSetPredicate newFieldCompareSetPredicate = new FieldCompareSetPredicate(
	OnlineFeeTaxFields.EffectiveDate, //Defines the "outer" field.
	null, 
	OnlineFeeTaxFields.EffectiveDate.SetObjectAlias("ft"), //Defines the "inner" query field. SetObjectAlias gives the table an alias, SetFieldAlias does the same for a field.
	null,
	SetOperator.Equal, //We are looking for equality.
	(
		OnlineFeeTaxFields.State == OnlineFeeTaxFields.State.SetObjectAlias("ft") & 
		OnlineFeeTaxFields.TransactionTypeId == OnlineFeeTaxFields.TransactionTypeId.SetObjectAlias("ft") & 
		OnlineFeeTaxFields.TransactionCompanyCode == OnlineFeeTaxFields.TransactionCompanyCode.SetObjectAlias("ft") &
		(
			OnlineFeeTaxFields.SecOrgId.SetObjectAlias("ft") == 4 | OnlineFeeTaxFields.SecOrgId.SetObjectAlias("ft") == -1)
	), //Above is a fairly complex "predicate" filtering the subquery
	null, //No relationship map
	"", //No name from the relationship map
	1, //We want the "top" 1 record.
	new SortExpression(OnlineFeeTaxFields.EffectiveDate.SetObjectAlias("ft") | SortOperator.Descending)); //Sort descending so the largest is on top

Here we have an example of *filtering* on a related table. Prefetch and filter are not directly related: we can filter on tables we don't fetch and fetch related tables we didn't filter on. Prefetched tables are filtered based on the relationships established in the database, so it isn't frequently a requirement to filter prefetched tables.

        public static EntityCollection GetCarriers(Guid programId)
        {
            EntityCollection carriers = new EntityCollection();
            PrefetchPath2 path = new PrefetchPath2(EntityType.CarrierEntity);
            path.Add(CarrierEntity.PrefetchPathCarrierProgram);
            RelationPredicateBucket bucket = new RelationPredicateBucket(
                CarrierProgramFields.ProgramGuid == programId);
            bucket.Relations.Add(CarrierEntity.Relations.CarrierProgramEntityUsingCarrierGuid);

            SortExpression sorter = new SortExpression(CarrierFields.CarrierName | SortOperator.Ascending);

            using (DataAccessAdapter adapter = new DataAccessAdapter())
                adapter.FetchEntityCollection(carriers, bucket, 0, sorter, path);

            return carriers;

        }

Category:

Hints and Tips

Small Business Backup Solutions

jlopez — Wed, 17 Aug 2005 17:56:52 +0000

This is one of those questions that should be asked about more often, "How do I protect my data". Many small businesses take unnecessary risks with their livelihood by ignoring the most basic concept of computing: the value of your data. Since most small businesses are based on Windows platforms, the information here is tinged with Windows terminology, but the ideas are generic enough to use in most environments.

Business owners understand the value of their data, otherwise there would be little need for a computer system. (Even if you only do word processing and a few spreadsheets, think of the difficulty of recreating that information. If you don't like that idea, then you need to create backups.) What is often not understood is the risk the data is placed in every day simply by using their computer system. Let us look at some common small business data environments to understand the risks and some ways to mitigate those risks.

Small company backups

In a very small business there may be only a single machine (this is frequently the case in small retail establishments) that does everything. This machine is responsible for point of sale, accounting, word processing and perhaps even web browsing and recreational use. Thus a failure of this computer means a total loss of electronic data, and a lack of backups could be a disaster (reconstructing finances from paper receipts is time consuming and error prone).

Now consider the fact that most small companies don't have IT technical assistance and so probably have an insecure environment that can be infected with malware, viruses and simply damaged by accidents such as deleting critical system folders or the data itself. Calls of the nature "I just lost my hard drive: could you get my data for me?" are a sadly all too common. The cost of such data recovery can be thousands of dollars if there are physical problems with the drive as the drive will need to be sent to a clean-room for recovery.

On the other hand, most companies of this nature have very small total quantities of data (perhaps around 100MB). This means that most modern computers come with a handy backup solution built into them: the CD-R or DVD-R drive. Recordable CDs can hold 650-750 MB of data or more and DVDs hold 4300 MB of data. Both types of media are available in bulk, CD-R costs around 12 cents each and DVD-R around 25 cents, at the time of this writing. Many of the drives come with backup utilities or "packet mode" writers (which allow you to treat the drive as a standard drive), both of which can be used effectively to back up your data. (If a machine doesn't come with a recordable drive, they are easily added for little cost).

With backup software the process can be automated so all that is necessary is place a fresh disk in the drive at the end of the day and allow the scheduled backup to copy your data to the disk. If you are using the packet mode writing feature, dragging and dropping your files on the disk will suffice. But which files should be backed up?

To make your backups as quick and easy as possible, it is best if you can organize your files into a single folder. Folders under this main location can be used to organize data for easy navigation. For a single computer user, this can be as simple as ensuring all documents reside under "My Documents" and backing that up. Some software may by default save files to other locations (such as the program directory itself) in which case it is best to check to see if the defaults can be changed.

One thing to keep in mind is that the files should not be open at the time the backup is made as most backup software will skip open files, which defeats the purpose of backing them up in the first place. When backing up databases this can be complex, as some database software runs as a "service" and the files are open as long as the service is running. Consult the product documentation for recommended methods of backing up any software that uses databases storage. Often a workable solution is to use the product's own backup tool to create a backup file in your main folders, which will in turn be backed up when you execute your normal backup procedures.

Rewritable and off site

It is important to take your backup media off-site to avoid loss of your information due to loss of the building itself. For small companies this can simply mean taking the recorded backups home with the owner, or better yet to a safe deposit box. Note that any backup media is temperature sensitive. Leaving it in extreme temperatures may destroy the usefulness of the backup, so take care during transport and storage that such extreme temperatures are not a potential problem. Also note that it is a good idea to test the recovery procedures from time to time. Bring an off-site stored disk back on site and verify that you can actually recover data successfully. Failure to test backup restoration is another all too common source of heartache, and if the first time you test the restoration is when you need it, a failure is no longer just a nuisance.

An added advantage to using CD-R and DVD-R media is that in the case of an audit you will have frequent, reliable snapshots of the state of the business. Data written to -R products is not alterable: the packet mode drivers that allow overwriting of data simply write new data and refer to it by default, but the old data remains on the disk.

However, it is wise to also consider the rewritable versions of the media you use: a couple of packages of rewritable disks makes it an option to run a daily backup Monday - Thursday (for example) on rewritable disks and on Friday on a write once media which is stored off site and acts as your archival media. Write once media has a longer shelf life and because of this rewritable media should not be your only optical backups. Over time the reuse of the rewritables will keep storage and costs down. Make sure to note the number of rewrites the media is rated for and use only a portion of those rewrites before replacing the media with fresh disks.

Network backups

A slightly more complex backup situation is the small network. Two or more computers may be in use and backing up all the data is just as important as it was with a single machine. For very small networks the easiest solution can be to use a single machine as the "server" for the other machines. This means you still store all of your data in a single folder structure, which is now shared among the users. It used to be that such arrangements were hazardous because of the stability problems with Windows 98 and earlier, but from Windows 2000 forward, the professional editions have been excellent choices for a small business to operate a peer to peer network with a central storage machine. In even larger companies (more than five machines) or for those who want the extra assurance of stability, purchasing a low end server machine that acts as a central storage machine is an excellent investment.

In either of these cases you may still be able to perform backups exactly as outline above, albeit the amount of data will increase. With DVD-R it is quite reasonable for a small company to still be able to backup to a single disk.

Another option (and one that can be quite effective) is to allow each of the users to store data locally, but to copy it to a central location (peer server or full server) periodically. This can be done as simply as a scheduled batch file running the "xcopy" command, via software designed for this purpose (such as Second Copy 2000) or by backup software that can target a file share for storage. This central location can then be backed up to recordable media, and can act as "first stop" for any data recovery. Specialized programs like Second Copy 2000 can be configured to maintain multiple versions of files, such as the last five revisions of a file in an archive, protecting against accidental changes being saved and overwriting legitimate data.

Larger requirements
Of course, eventually your data may exceed the capacity of even DVD-R, in which case a more traditional backup solution may be required. Another article will be posted covering medium weight backup solutions in the near future. Some may be wondering why tape backup has not been mentioned: that is because for the light weight needs of very small business, tape can be more expensive and complex than the simple solution outlined here. The article on medium weight backup solutions will consider tape's pros and cons more fully.

Category:

Hints and Tips

CutePDF

jlopez — Wed, 27 Jul 2005 18:26:42 +0000

Many people need to generate PDF files occasionally, but it can be an expensive proposition to get the latest version of Adobe Acrobat for every user in your office to create basic PDF files. Acrobat comes with many wonderful features for create sophisticated PDF files. Digital signing, encryption and forms that post to web servers are among a few that Timewarp Technologies uses Acrobat for. However, when all you need is get some pages from some unusual program and provide the results to a user without the ability to read the source or resulting files, PDF is a great solution. For that matter, if the end consumer should not be modifying the document, I highly recommend sending PDF instead of Word documents as they are safer and more robust.

Enter CutePDF Writer:

http://www.cutepdf.com/Products/CutePDF/writer.asp

This handy program will allow you to "print" to the CutePDF printer and save the result as a PDF you can then send along to your end consumer. Simple, powerful and best of all, free.

Free? Why free? Well, CutePDF uses Ghostscript, an open source (and itself free) postscript manipulation program as the conversion tool. The company provides more powerful tools that build upon CutePDF which are all quite affordable as well, but the print to PDF functionality remains a free product that does everything the average user will ever need. If you do need additional functionality, try the companies affordable CutePDF product:

http://www.cutepdf.com/Products/CutePDF/plus.asp

Both are highly recommended. Of course, if you need the most advanced PDF features only Acrobat will do:

http://www.adobe.com/products/acrobatpro/main.html

Category:

Hints and Tips

Why is NAT good for you?

jlopez — Tue, 05 Jul 2005 20:23:10 +0000

Recent articles have pointed to a 50% chance that an "unprotected Windows PC" will be infected within 12 minutes of being connected to the Internet.

Globe and Mail article on exploits of Windows Systems

What this means is that if a machine is installed with Windows and then goes online to obtain the necessary security updates, it is probably exploited before completing the update process, with all the negative consequences thereof. Windows XP with Service Pack 2 is already partially "hardened" compared to prior versions of Windows, which makes a most compelling argument for updating to the newer operating system if it is possible when it becomes necessary to reinstall windows. However, it is often necessary to reinstall Windows due to some calamity that befell the operating system, and such reinstalls usually will be with the existing operating system the machine is licensed for. If you only have 12 minutes before having a 50/50 chance of being exploited, how in the world can you safely update it?

One way is to use "slipstream" updates or manually installing updates that were obtained by other means, but this is a complex and confusing undertaking for the average computer user. Is there a simpler way?

It turns out there is: use a NAT capable router. Almost all "broadband" routers come with NAT or Network Address Translation. NAT isn't a "true" firewall, but it eliminates nearly all attack vectors that Windows is normally exposed to during the update process. In addition, they are incredibility easy to set up, which can't be said of "true" firewalls. (On the other hand, if you many machines or special security needs, NAT can't replace the capabilities of the true firewall products, so this advice is mostly for those with home computers.)

If you have a machine behind a NAT router (or even better, one a true firewall), you can safely use the Windows update service to bring your machine up to date and secure it against the known vulnerabilities. What's better is that the NAT functionality will protect against the "remote exploit" vulnerabilities that come to light in the future.

A NAT capable broadband router should cost $50-$100 for a 4 port version, which gives the additional benefit of allowing more than one machine to share a single broadband connection. It is a small investment when you consider the cleanup costs of an exploited machine.

Note that a NAT capable device or a firewall is not a magical solution... it will not protect against viruses, Trojans and malware that arrive from websites that you download programs from (intentinally or unintentionally) or e-mail. The best defense is a layered defense, which means a NAT capable device or firewall, a "software" firewall (a limited one is provided in Windows XP Service Pack 2) and an anti-virus program. The more layers of defense, the less likely any one failing will cause your machine to be exploited. Note though that third party software firewalls and anti-virus programs must be kept up to date to remain effective and not become a potential path of exploitation themselves. These products will be discussed in a later article.

Froogle Search on "Broadband Router"

Category:

Hints and Tips