Django South Notes

Posted on by
Reply

South

South is “a tool to provide consistent, easy-to-use and database-agnostic migrations for Django applications.” However, as South’s website says, since Django 1.7 now has its own migrations tool South has been deprecated. I personally think the new Django migrations are a little bit better.

However we still use South at FiveStars and I thought other people who still use South might appreciate these notes.

See list of migrations (asterisks means already applied)

[shell]
$ ./manage.py migrate –list [app name]
[/shell]

Apply latest migrations

[shell]
$ ./manage.py migrate [app name]
[/shell]

Fake migration

[shell]
$ ./manage.py migrate [app name] [version #] –fake
[/shell]

Migrate backwards

[shell]
$ ./manage.py migrate [app name] [version #]
[/shell]

Migrate back to zero state, i.e. nothing

[shell]
$ ./manage.py migrate [app name] zero
[/shell]

Create first migration

[shell]
$ ./manage.py schemamigration [app name] –initial
[/shell]

Create subsequent migrations

[shell]
$ ./manage.py schemamigration [app name] —-auto
[/shell]

Create empty schema migration

[shell]
$ ./manage.py schemamigration [app name] —-empty [migration name]
[/shell]

Create data migration

[shell]
$ ./manage.py datamigration [app name] [migration name]
[/shell]

Review migration history

[sql]
select * from south_migrationhistory order by applied;
[/sql]

Redo migrations that have been hand edited

Sometimes you will find someone followed the bad practice of hand editing a migration after it has been committed. For example they’ll add another column. Because you have already run the migration you may not notice this until many migrations later when something breaks because of this missing column.

To fix this:

  1. Fake migration backwards to the edited migration
  2. Migrate back one before the edited migration
  3. Migrate forward one for the edited migration
  4. Fake migrate forwards the rest until you get to your current state
  5. Migrate the migration that previously was giving problems

Regular Expressions in Oracle SQL

Posted on by
Reply

I Know Regular Expressions

For a project I needed to change the picture URL’s of all users. Fortunately Oracle SQL supports regular expressions making this problem much easier to solve.

SQL to select pictures to change:

SELECT picture FROM users
WHERE REGEXP_LIKE(picture, '^graph\.facebook\.com.*?type=square');

SQL to update pictures from type=square to type=large:  

UPDATE users SET picture =
REGEXP_REPLACE(picture,
               '^(graph\.facebook\.com.*?type=)square$', '\1large')
WHERE REGEXP_LIKE(picture, '^graph\.facebook\.com.*?type=square$');

SQL to select updated pictures:

SELECT user_id, picture FROM users
WHERE REGEXP_LIKE(picture, '^graph\.facebook\.com.*?type=large$');

For more information please read Using Regular Expressions in Oracle Database.

Don’t always use GenericService

Posted on by
Reply

Sorry... what exactly is in this can again?

Sorry… what exactly is in this can again? by Paul Jerry, on Flickr

This is the most common thing you will see in an ATG Java class.

public class Foo extends GenericService

However 90% of the time it is extending GenericService so that the class has access to ATG’s logging capabilities.

This is overkill. The GenericService is much more than a logging service. As its name suggests it is meant to be extended by services, typically globally scoped. The main reason to extend GenericService is to get access to methods for controlling the service like doStartService and doStopService.

If you just want logging instead extend ApplicationLoggingImpl. It’s lighter weight and it’s clear why you are extending it. The only caveat is you have to remember to set the loggingIdentifier property in the properties file. For example:

# /betweengo/Foo
$class=com.betweengo.Foo
loggingIdentifier=betweengo.Foo

Another advantage of using ApplicationLoggingImpl is that the source is available in DAS/src/Java in the ATG installation unlike with GenericService.

Hopefully some people will read this post and on my next consulting project I won’t see such egregious uses of GenericService. Smile

ATG Ordered One-to-Many Relationships

Posted on by
Reply
Former NY Knicks guard Jeremy Lin wins debut with Houston Rockets as James Harden explodes for 37 points

Former NY Knicks guard Jeremy Lin wins debut with Houston Rockets as James Harden explodes for 37 points

Last week I blogged about ATG unordered one-to-many relationships. Ordered one-to-many is quite similar but a little more complex.

Here again is the same example of players belonging to a team except now the players are a list property of the team.

[code language=”xml”]<item -descriptor name="team">
<table name="team" type="primary" id-column-names="id">
<property name="name" required="true"></property>
</table>
<table name="player" type="multi" multi-column-name=”idx” id-column-names="team_id">
<property name="players" column-name="id" data-type="list" component-item-type="player"></property>
</table>
</item>

<item -descriptor name="player">
<table name="player" type="primary" id-column-names="id">
<property name="name" required="true"></property>
<property name="team" column-name="team_id" item-type="team"></property>
</table>
</item>[/code]

And here is the SQL.

[code language=”sql”]CREATE TABLE team (
id VARCHAR2(40) NOT NULL,
name VARCHAR2(3) NOT NULL,
CONSTRAINT team_p PRIMARY KEY (id)
);

CREATE TABLE player (
id VARCHAR2(40) NOT NULL,
idx NUMBER(2,0),<p> name VARCHAR2(40),
team_id VARCHAR2(40),
CONSTRAINT player_p PRIMARY KEY (id)
);[/code]

Now comes the tricky part. You can never just set the team in the player repository item.

[code language=”java”]player.setPropertyValue("team", team);[/code]

This is because when you do this the idx column in the player table is not set. We could create a repository property and try to set it separately but that is bad practice. You would have to know how many players are already on the team and then set the idx to that number.

What you should do is not allow the team property to be written.

[code language=”xml”]<property name="team" column-name="team_id" item-type="team" writable="false" />;[/code]

And instead add the player to the team repository item.

[code language=”java”]Set<RepositoryItem> players = (Set<RepositoryItem>) team.getPropertyValue("players");
players.add(player);[/code]

Because of this requirement you cannot make the team property required for the player as discussed before.

Again if you have any questions please leave them in the comments.

ATG Unordered One-to-Many Relationships

Posted on by
Reply
Bautista, Lawrie, Rasmus

Photo by james_in_to

Previously I blogged about specifying one-to-many relationships in ATG. Turns out I could have done it much more simply.

Using the same example of players belonging to a team here are the simplified SQL repository definitions.
[code language=”xml”]<item-descriptor name="team">
<table name="team" type="primary" id-column-names="id">
<property name="name" required="true" />
</table>
<table name="player" type="multi" id-column-names="team_id">
<property name="players" column-name="id" data-type="set" component-item-type="player" />
</table>
</item-descriptor>

<item-descriptor name="player">
<table name="player" type="primary" id-column-names="id">
<property name="name" required="true" />
<property name="team" column-name="team_id" item-type="team" />
</table>
</item-descriptor>[/code]
And here is the SQL.
[code language=”sql”]CREATE TABLE team (
id VARCHAR2(40) NOT NULL,
name VARCHAR2(3) NOT NULL,
CONSTRAINT team_p PRIMARY KEY (id)
);

CREATE TABLE player (
id VARCHAR2(40) NOT NULL,
name VARCHAR2(40) NOT NULL,
team_id VARCHAR2(40),
CONSTRAINT player_p PRIMARY KEY (id)
);[/code]
You can set the team in the player repository item.
[code language=”java”]player.setPropertyValue("team", team);[/code]
Or add the player to the team repository item.
[code language=”java”]Set<RepositoryItem> players = (Set<RepositoryItem>) team.getPropertyValue("players");
players.add(player);[/code]
What gets tricky is if you require a player to be on a team.
[code language=”xml”]<property name="team" column-name="team_id" item-type="team" required="true" />[/code]
[code language=”sql”]team_id VARCHAR2(40) NOT NULL,[/code]
In this case you cannot remove a player from a team by doing this:
[code language=”java”]Set<RepositoryItem> players = (Set<RepositoryItem>) team.getPropertyValue("players");
players.remove(player);[/code]
This is because when you remove the player from the team the player will no longer have a team. This violates both the repository definition and will result in a SQL error because team_id cannot be null.

The only way you can change a player’s team is to do this.
[code language=”java”]player.setPropertyValue("team", newTeam);[/code]
To help enforce this you can make the team’s players property unwritable.
[code language=”xml”]<property name="players" column-name="id" data-type="set" component-item-type ="player" writable="false" />[/code]
I hope this makes sense. If you have any questions please leave them in the comments.

Ignore Unavoidable Generic Type Problems in Eclipse

Posted on by
Reply

When you use Eclipse by default most generic type problems show up as warnings. Unfortunately this results in lots of warnings when you use ATG code.

You can turn off the warnings that result from ATG code by checking the option “Ignore unavoidable generic type problems”.

Ignore unavoidable generic type problems

Ignore unavoidable generic type problems

For example you might have code like this where getValues() comes from the ATG API:

getValues().put(name, value);

By default Eclipse reports warnings for the above line. But this is unavoidable and with this Ignore option turned on no warning is reported for this line.

Kind of useful and you don’t have to add @SuppressWarnings(“unchecked”) liberally through your ATG code.

Debugging Transaction is not Active

Posted on by
Reply

Problems like this happen occasionally and are quite bedeviling.

Caused by: java.lang.RuntimeException: CONTAINER:atg.repository.RepositoryException; SOURCE:org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >)
    at atg.adapter.gsa.GSAItemDescriptor.loadProperty(GSAItemDescriptor.java:5479)
    at atg.adapter.gsa.GSAItem.getPersistentPropertyValue(GSAItem.java:1101)
    at atg.adapter.gsa.GSAItem.getPropertyValue(GSAItem.java:994)
    at atg.adapter.gsa.GSAItem.getPropertyValue(GSAItem.java:1272)
    at atg.repository.RepositoryItemImpl.getPropertyValue(RepositoryItemImpl.java:128)
    at atg.commerce.order.processor.ProcLoadHandlingInstructionObjects.runProcess(ProcLoadHandlingInstructionObjects.java:183)
    at atg.service.pipeline.PipelineLink.runProcess(PipelineLink.java:233)
    at atg.service.pipeline.PipelineChain.runProcess(PipelineChain.java:343)
    ... 17 more
Caused by: CONTAINER:atg.repository.RepositoryException; SOURCE:org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >)
    at atg.adapter.gsa.GSAItemDescriptor.loadProperties(GSAItemDescriptor.java:5431)
    at atg.adapter.gsa.GSAItemDescriptor.loadProperty(GSAItemDescriptor.java:5471)
    ... 24 more
Caused by: org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >)
    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:96)
    at atg.service.jdbc.WatcherDataSource.getConnection(WatcherDataSource.java:801)
    at atg.service.jdbc.WatcherDataSource.getConnection(WatcherDataSource.java:782)
    at atg.adapter.gsa.GSATransaction.getConnection(GSATransaction.java:744)
    at atg.adapter.gsa.GSAItemDescriptor.getConnection(GSAItemDescriptor.java:2365)
    at atg.adapter.gsa.GSAItemDescriptor.loadProperties(GSAItemDescriptor.java:5345)
    ... 25 more
Caused by: javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: 7f000001:c525:503e32fb:2f2c status: ActionStatus.ABORT_ONLY >
    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:319)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:403)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:850)
    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:90)
    ... 30 more

I tweeted about issues like this over two years ago.

As I said in the tweet this almost always is not a transaction issue but actually an application issue. The tweet references an excellent article Transaction is not active: tx=TransactionImple < ac, BasicAction in which the author found in his case that the problem was a NullPointerException which caused the transaction to end.

Today I had to debug this issue again. First I checked the transaction in ProcLoadHandlingInstructionObjects before it called getPropertyValue on the shipping group repository item.

GSAItem item = (GSAItem) sgMutItem;
ItemTransactionState state = item.getItemTransactionState(false);
logDebug("state=" + state.mGSATransaction);

I saw in the logs that the status of the transaction was ActionStatus.ABORT_ONLY which meant the transaction was already aborted before getting the property value from the repository which is why the Transaction is not active exception happened.

Next I checked the transaction at the beginning of ProcLoadHandlingInstructionObjects using the order repository item.

GSAItem item = (GSAItem) orderItem;
ItemTransactionState state = item.getItemTransactionState(false);
logDebug("state=" + state.mGSATransaction);

When I confirmed that the status of the transaction was ActionStatus.ABORT_ONLY at the beginning of the processor I went backwards through the pipeline checking all the processors in the same way.

After doing that and confirming the transaction was ActionStatus.ABORT_ONLY at the beginning of the pipeline process I began checking the code that called the pipeline using code like this.

GSAItem orderItem = (GSAItem) order.getRepositoryItem();
ItemTransactionState state = orderItem.getItemTransactionState(false);
try {
  int status = state.mGSATransaction.getTransaction().getStatus();
  if (status == Status.STATUS_MARKED_ROLLBACK) {
    logError("Oh-oh, marked for rollback.");
  }
} catch (SystemException exc) {}

I finally found the problem was that in a previous call to another pipeline an error had occurred and the transaction was marked for rollback. However the code did not check the result of the pipeline call and had continued.

The moral of the story is always check your return values and don’t go down the rat hole of believing it’s a transaction issue.

Alice in front of the rabbit hole

Exporting ATG Repositories

Posted on by
2
Kmtc Jakarta at singapore | Flickr

Kmtc Jakarta at singapore by wirralwater

ATG’s documentation suggests running startSQLRepository to export repositories. However this is an older solution that works well with SOLID and DAS but not so easily with Oracle and JBoss. Oracle uses its own JDBC jar which you have to add to the CLASSPATH before running startSQLRepository. JBoss uses JNDI and real DataSource’s to connect to the database. But this doesn’t work with startSQLRepository so you have to set up a separate ATG server with fake DataSources to run with startSQLRepository.

All of this is to say it’s a pain to use startSQLRepository. The much simpler method is to navigate to the repository in your Dynamo admin (e.g. http://localhost:8080/dyn/admin). Then in the “Run XML Operation Tags on the Repository” textbox enter the command
[xml]<export-items />[/xml]
Or if you want to export only certain item-descriptors:
[xml]<export-items item-descriptors="authors, books" />[/xml]
And if you want to ensure you only get a certain item-descriptor without any referenced item-descriptors:
[xml]<export-items item-descriptors="books" skip-references="true" />[/xml]
That Was EasyYou can then save the outputted XML to an XML file and then later paste the contents of this XML file to the same textbox to import the repository items.

For more information please see the <export-items> section in the SQL Repository Reference chapter of the ATG Repository Guide.

Why is there a _requestid in my URL?

Posted on by
Reply
Tree Tutorial | Flickr

Tree Tutorial by eilonwy77

Sometimes when you are using an ATG powered website, e.g. Keds, you will see a _requestid parameter in your URL, e.g. http://www.keds.com/?_requestid=138580.

This occurs when the ATG code, typically a form handler, does a local redirect instead of a redirect. This is ATG’s own flavor of redirect which does a few extra things as described in the JavaDoc:

Sends a redirect response to the client using the specified redirect location URL. This function is similar to sendRedirect(), with the following two differences:

  • The session ID is added to the URL if a valid cookie with the session ID is not found.
  • If the URL is a relative URL (i.e., “login/error.html”), then the URL will be converted to a full absolute URL, as required by the HTTP specification. A relative URL is one that does not specify a protocol (e.g., “http:”).

In general, this function should be used when redirecting back to a page on the same site. If you are redirecting to a page on some other site, use the full absolute URL and call sendRedirect().

However this does not explain the mysterious _requestid parameter. Fortunately ATG’s Programming Guide does:

Preserving Request Scoped Objects on Redirects

If a request results in a redirect to a local page through the method HttpServletResponse.sendLocalRedirect(), the ATG platform treats the redirect request as part of the original request, and maintains any request-scoped objects associated with that request. To implement this, the ATG platform adds an additional query parameter named _requestid to the redirected URL.

Now you know what the _requestid parameter is and why it exists. And if you don’t want it then you the programmer should use a redirect instead of a local redirect.