If you haven't had a chance to work with it, consider visiting

http://dlc.sun.com/virtualbox/vboxdownload.html

It runs on windows, linux, mac os/x and solaris.

I downloaded a pre-installed VM with a database, development tools, and an application server pre-installed.  Though I had the minimum required config (~2 GHz processor, 2 GB RAM), I was surprised that it ran as well as it did.

Though the initial setup is quite quick, the downloads of actual VMs to use in VirtualBox can take a while, but compared to the aggravation of pulling together Oracle 11gR2, SQL Developer, Apex, and a Weblogic server on your own, it's a more than fair trade.

If you've been putting off exploring Hands-On Database Technology on a home PC or a spare machine at work (http://www.oracle.com/technetwork/database/enterprise-edition/databaseappdev-vm-161299.html), then this could be a good way to jump in — with a virtual machine.

Anyone who's used SQL Developer may have noticed the search engines tucked away in the upper right corner of the screen.  With a nod to Firefox, the people who created SQL Developer made it easy to look up an error message on Google, consider going to Oracle Technology Network (OTN) forums with your SQL Developer issue, or checking the AskTom or Oracle documentation sites.  

While the sites provide valuable information, often they get into esoteric areas or long blogs.  PSOUG succinctly exposes the concepts that a developer needs right away with examples of usage and scripts that illuminate topics such as CREATE MATERIALIZED VIEW.

It's exceedingly easy to try out.  Add PSOUG to your search domains by placing an image file and a source file in the sqldeveloper/sqldeveloper/searchengines path of your installation.

First file: psoug.src

<search
   name="PSOUG"
   description="PSOUG Search"
   method="POST"
   action="http://psoug.org/search.htm"
   searchForm="http://psoug.org/search.htm"
   queryEncoding="utf-8"
   queryCharset="utf-8"
>

<input name="qwords" user>
<input name="querytype" value="allwords">
<input name="catid" value="0">
<input name="searchtype" value="basic">
<input name="submit" value="GO">

<interpret
    browserResultType="result"
    charset = "UTF-8"
    resultListStart="<!–a–>"
    resultListEnd="<!–z–>"
    resultItemStart="<!–m–>"
    resultItemEnd="<!–n–>"
>
</search>

Second file: psoug.gif

I downloaded the PSOUG title.gif and scaled it down by a factor of about 4.  Use whatever tool makes sense to manipulate your image such as GIMP (multi-platform)  or Microsoft's Image editor, or whatever works on the Mac OS/X into an icon-sized image.  Jpeg, png, gif are all acceptable formats.

Your files will be discovered automatically the next time you start SQL Developer.

A shell database is essentially the logical starting point of a database wth some tables preloaded with essential lookup data, stored database code, constraints, and views defined, but the many of the segments start out empty.

To build a new database deliverable one can follow a number of steps.

1)  Get the latest shell (export dmp)  and create user SQL scripts from version control

2) Connect to the target database with SYSDBA and drop the application users (USERA, USERB)  and their tablespaces (including USER_DATA and USER_INDEXES).

3) Create the users' tablespaces, then create the users anew with the SQL scripts.

4) Import the latest shell  (e.g.  imp fromuser=(USERA,USERB)  touser=(USERA,USERB) file=last_shell.dmp log=step4.log )

5) Apply any database upgrade scripts readied for deployment since last shell was created.

6) Drop and recreate sequences.  This will assure that the actual sequence numbers will be computed from values in associated tables.

7) Review the newly created schemas and recompile any invalid triggers, PL/SQL, and views.   Investigate objects that won't recompile and remove any obsolete objects not cleared by upgrade scripts. Invalid objects will raise errors when imported by end-users and cause calls to product technical support.

8) Rebuild indexes on unique or primary keys that appear to be on the USER_DATA into the USER_INDEXES tablespace.

9) Export the full database  — save this as it will be needed later.  (e.g. exp userid=sys file=NEW_SHELL_STEP9.dmp full=Y compress=N log=STEP13.log )

10) Export the database — all user object definitions
         (e.g. exp userid=USERA file=NEW_SHELL_USERA.dmp  rows=N statistics=NONE log=STEP10A.log )

         (e.g. exp userid=USERB file=NEW_SHELL_USERB.dmp  rows=N statistics=NONE log=STEP10B.log )

11) Export the database — all user data

         (e.g. exp userid=USERA file=NEW_DATA_USERA.dmp  rows=Y statistics=NONE log=STEP11A.log )

         (e.g. exp userid=USERB file=NEW_DATA_USERB.dmp  rows=Y statistics=NONE log=STEP11B.log )

12) With SQLPlus, connect to the target database as SYSDBA, then enter the following commands:

        DROP USER USERA CASCADE;

        DROP USER USERB CASCADE;

      then re-run the create users script. 

13) Extract DDL for creating USERA objects to a text file.
         (e.g. strings NEW_SHELL_USERA.dmp  | egrep -v -fgrep_v.inp | sed -fsed_nl.inp > USERA_DDL.SQL )

14) Extract DDL for creating USERB objects to a text file.
         (e.g.  strings NEW_SHELL_USERB.dmp  | egrep -v -fgrep_v.inp | sed -fsed_nl.inp > USERB_DDL.SQL )

15)  Edit the USERA_DDL.SQL to remove extraneous information
           Remove the header information from the 1st line down to the line after CONNECT USERA.
           Skip DDL statements until you find the first CREATE TRIGGER, CREATE FUNCTION,  CREATE PROCEDURE, or CREATE VIEW statement.  Delete all lines from this point to the end of the file.

15) Perform the same editing to the USERB_DDL.SQL file.

16) With SQLPlus, connect to the target database as USERA, then enter the following commands:

           SPOOL STEP16.LOG
           SET ECHO ON
           @USERA_DDL.SQL
           SPOOL OFF
           EXIT

17) With SQLPlus, connect to the target database as USERB, then enter the following commands:

           SPOOL STEP17.LOG
           SET ECHO ON
           @USERB_DDL.SQL
           SPOOL OFF
           EXIT

18) Re-import the data saved from Step 11.

  (e.g. imp userid=usera file=DATA_USERA.dmp log=STEP18A.log ignore=Y rows=Y constraints=N )

  (e.g. imp userid=userb file=DATA_USERB.dmp log=STEP18B.log ignore=Y rows=Y constraints=N )
 

19) Re-import the data saved from Step 10

  (e.g. imp userid=sys fromuser=USERA touser=USERA file=NEW_SHELL_USERA.dmp log=STEP19A.log ignore=Y rows=N constraints=Y )
  (e.g. imp userid=sys fromuser=USERB touser=USERB file=NEW_SHELL_USERB.dmp log=STEP19B.log ignore=Y rows=N constraints=Y )
 

20) Export data for the last time

 (e.g. exp userid=sys full=Y file=last_shell.dmp compress=N log=STEP20.log )

21) Put the last_shell.dmp back into your source control system.

================================================
NOTES: Step 13, Step 14:     The strings command is available on both Windows and Unix platforms.
================================================

It might be a good idea to store the extracted DDL files in version control as they are useful artifacts of the software development process and are useful as reference representations of the database definitions at various points in the software lifecycle.
 

Instead of upgrading an Oracle database in place, some plans also call for moving to new hardware. A tried and true approach is to generate scripts and use DataPump exports to recreate your application data on the future database. 

Let me tell you about of a team of DBAs who took an export and began to import the data structures and data rows into the new database.  After a while, they realized one of the schemas was kind of large and that the whole operation  might take longer to complete than allotted.  Desparate to beat the clock, they began reverse engineering all the tables schema by schema and applying the resulting scripts to the new target database.  Now DBAs were competing with the import to create schemas and move data.

Triggers were not firing in the new system, and applications fell flat on their faces.  Alas, the grants, synonyms, roles, and sequences were left behind in this haste, and the applications were not working on the new production environment.

What to do to rescue the installation?  Well, reverse engineering sequences would be an onerous chore with the shiny, trusted front-end tool that the DBAs used every day because navigating to each application schema and manually clicking on each sequence to reveal its definition. 

The other issue was not ALL of the sequences were needed.  Not all of those SYS, sample, special purpose schemas created by Oracle installations were going to be migrated.  Besides the new production database was all set and we didn't want collisions or kibble.

The solution:  use dynamic SQL to generate a single script of the sequences required from the old system, and run them on the new system, excluding unnecessary schemas.

Here's the Oracle SQL script that took care of that problem:

          select    
          'CREATE SEQUENCE '||sequence_owner||'.'||sequence_name||'  START WITH '|| TO_CHAR(last_number+1) ||
          ' INCREMENT BY '|| TO_CHAR(increment_by)||' MAXVALUE '||TO_CHAR(max_value) ||  
          ' MINVALUE '||TO_CHAR(min_value),
           CASE WHEN cycle_flag='N' THEN ' NOCYCLE ' ELSE ' CYCLE ' END cyc_flag,
           CASE WHEN order_flag='N' THEN ' NOORDER ' ELSE ' ORDER ' END ord_flag,
–          sequence_owner, sequence_name, last_number,
           CASE WHEN  cache_size=0 THEN  ' NOCACHE ' ELSE ' CACHE '||TO_CHAR(cache_size) END,
           ' ;'
–                     min_value, max_value,  
–                     increment_by,
–                     cycle_flag,
–                     order_flag
             from sys.dba_sequences
           where  
                 sequence_owner NOT IN ('SYS','SYSTEM') — usual DBA accounts
             and sequence_owner NOT IN ( — Oracle 9i Demo Schemas
                                 — Doc ID:     207560.1     Type:     FAQ      Modified Date:     22-MAR-2009
                          'HR',–        (Human Resources)
                          'OE',–        (Order Entry)
                          'PM', –       (Product Media)
                          'SH',–        (Sales History)
                          'QS',–        (Queued Shipping)
                          'QS_ADM',–    (Administration)
                          'QS_WS',–     (Western Shipping)
                          'QS_ES',–     (Eastern Shipping)
                          'QS_OS',–     (Overseas Shipping)
                          'QS_CBADM',–  (Customer Billing Administration)
                          'QS_CB',–     (Customer Billing)
                          'QS_CS')–     (Customer Service)
              and sequence_owner NOT IN ('IX',  — new sample schema in Oracle 10g
                                 'BI')
              and sequence_owner NOT IN (– Schemas created by Oracle
                                 — Doc ID:     160861.1     Type:     REFERENCE      Modified Date:     29-MAY-2009
                          'SCOTT', –  by script $ORACLE_HOME/rdbms/admin/utlsampl.sql
                          'OUTLN',–  by script $ORACLE_HOME/rdbms/admin/sql.bsq
                                                    — Optionally:
                            'DBSNMP',–                    if Enterprise Manager Intelligent Agent is installed  
                            'MGMT_VIEW',–                 is part of the DB Control Repository
                            'SYSMAN',–                    is part of the DB Control Repository and Grid
–                                                      see Note 270516.1 for details
                            'TRACESVR',–                  if Enterprise Manager is installed
                            'AURORA$ORB$UNAUTHENTICATED',– \
                            'AURORA$JIS$UTILITY$',–         — if Oracle Servlet Engine (OSE) is installed
                            'OSE$HTTP$ADMIN',–             /
                            'MDSYS',–                     if Oracle Spatial option is installed  
                            'MDDATA',–                    if Oracle Spatial option is installed  
                            'ORDSYS',–                    if interMedia Audio option is installed   
                            'ORDPLUGINS',–                if interMedia Audio option is installed   
                            'SI_INFORMTN_SCHEMA',–        if interMedia option is installed   
                            'CTXSYS',–                    if Oracle Text option is installed
                            'WKSYS',–                     if Oracle Ultra Search option is installed
                            'WKUSER',– (9i)
                            'WK_TEST',– (10g) if Oracle Ultra Search option is installed
                            'REPADMIN',–                  if Replication Option is installed  
                            'LBACSYS',–                   if Oracle Label Security option is installed  
                            'DVF',–                       if Oracle Database Vault option is installed
                            'DVSYS',–                     if Oracle Database Vault option is installed
                            'ODM',–                       if Oracle Data Mining option is installed
                            'ODM_MTR',–                   idem
                            'DMSYS',– (10g)               if Oracle Data Mining 10g option is installed
                            'DMSYS',–                     in 10g version to replace ODM and ODM_MTR schemas
                            'OLAPSYS',–                   if OLAP option is installed
                            'WMSYS',–                     if Oracle Workspace Manager script owmctab.plb is
–                                                      executed.
                            'ANONYMOUS',–                 if catqm.sql catalog script for SQL XML management
                            'XDB',–                       is executed
                            'EXFSYS',–                    is the Expression Filter Feature repository
                            'DIP',–                       for provisioning event processing
                            'TSMSYS'–                    Transparent Session Migration
                          )
                     order by sequence_owner, sequence_name ;

If other users besides the named owners need SELECT privileges on the sequence objects, then another script to generate those privileges would need to follow the sequences creation script, but that is an easy adaptation of the above script for most DBAs.

Dear DBA, as I am running loadtests with product XYZ in preparation
for uploading of many thousands of XML documents , I find that there is something
that's bogging down your database. I thought your equipment
should easily outpace the old Compaqs that we have running Windows, and that
Oracle should do better than our PostgreSQL-based setup. Yet
I find the numbers show the opposite:

733 Documents in 2325.713 s for an average of 3.173 s
at ~11% CPU load

944 Documents in 1960.4 s for an average of 2.077 s
at ~15% CPU load

It takes 50% longer on the Oracle setup. Since the
application server load is negligible at the same time, it
goes to show that we are waiting for the database.

I wonder why.

Hmmm, I thinking that this developer is not acquainted with the Oracle database technology choices available to him. He continues on.

Since I can not observe the Oracle database server's IO and
CPU activity directly, I have looked at the Oracle ADVISOR
report [Query 2: ADVISOR REPORT below]. I am showing
the most important part from report on task_id 4313
and task_name ADDM:6344221675_1_1349:
|
| FINDING 1: 60% impact (489 seconds)
| ———————————–
| Waits on event "log file sync" while performing COMMIT and ROLLBACK
| operations
| were consuming significant database time.

Is there anything you could do about this? For example, could
you move the redo logs onto a different disk device? Or is this
database or hardware very busy processing other transactions?

The loading of data might take at least 50% longer than expected
(and possibly more, because we'll hit the database with 4-8
processes in parallel). Your help will be much appreciated.

regards,
Prince

Dear Prince-
We've noticed many (hundreds of thousands) insert statements being executed through jdbc-client and wondered what your commit strategy was.

Sometimes java programs have auto commit turned on (every DML statement is committed one-at-a-time). If that's the case, then it is causing a lot of round-trips through the network to confirm a committed row in the database.

Other times, a bulk load will attempt to load thousands of rows at a time before committing. That can cause the growth of a large amount of undo segments that slows throughput as well. If that's the case then the commit interval needs to be changed.

If you are using jdbc's addBatch and executeBatch methods for processing multiple SQL statements, then you can make adjustments in how many rows in the batch that you need to commit.

Alternatively, you could make modifications to commit pending transactions after a given number of iterations in your code. If possible, try starting at 500 and varying it either higher or lower to meet your throughput goals.

One thing you should know about the test database that we're hosting is that we are not collecting and archiving redo logs.

We circulate through the logs very frequently. We have enlarged the redo log size to 200 MB (up from 55 MB), but we need to reduce the amount of redo being written. To that end, we recommend that the
insert SQL statements for the following tables be rewritten to use the "Append" hint since it appears that you are performing bulk load inserts on them.

If you want, we can even put the target tables into "nologging" mode during this phase where you need to load a lot of data. That would ensure that you could put hundreds of thousands of rows in the database in a very brief period of time.

1) Do you plan to

a) parse the XML docs and store them into multiple relational tables   ?

If so, what languages or techniques do you intend to use to accomplish this?

b) store an XML doc in a single XMLType column in the Oracle database?

Do you need to index the data to perform XML or "free text" searches?

If so, what XPATH indexing do you intend to perform?

c) Register XSD as an XML Schema and have Oracle validate incoming XML documents such that
data is automatically shredded into object-relational tables based upon the different
types and attributes?

If you choose (c), how will you reload and revalidate data against the new XSD should
new data elements become available in the XSD?

2) How important is it to preserve the exact XML document ingested in Oracle?

About Character Large Objects (CLOBs)

Yes, you can store CLOBs in line with a database row up to 4,000 characters, else it's written to a separate LOB segment.

Oracle automatically creates LOB indexes whether you want them or not, so you might want to review the non-XML CLOBS in your design and determine whether they would be well under the 4,000 characters and thus fit into a VARCHAR2 data column.

PSOUG.org is a great resource that people should be looking at when solving data management problems in Oracle.  Sharing knowledge and assisting others is part and parcel of the user group experience.

Although  a new feature from Oracle holds great promise, some strategies work better than others.  Developers and architects should know about the different ways and choose carefully to achieve better successes with their projects.

The spirit of Arkytekture is to share some real-world experiences (ok, maybe thinly disguised ones) that illustrate either database technology issues, software engineering practices, or some other philosophy.

Upcoming topics will include bulk-loading, measurements, error-handling, dynamic PL/SQL, and Apex.