Sunday, April 22, 2007

Collaborate07: my definitive review

The following is a collection of notes and reflective thoughts on the sessions I attended at IOUG's Collaborate07. Note that I am neither a writer or a speaker. =) Any mistakes, whether in form or content, are my own and should not be attributed to the session or author. However, I would ask that if you do spot any inaccuracies, please drop me a note or a comment so 1) I can learn and 2) make the correction.

Outstanding questions
  • Can Oracle TimesTen be used in conjunction with RAC? I hear that TimesTen will be compatible/integrated with 11g, but not exactly sure what that means.
  • With the introduction of SecureFiles, are LOBs being phased off the long-term roadmap?


Monday, April 16th, 8:am
General Session

Ari Kaplan kicked things off, but quickly turned things over to Ken Jacob (Dr. DBA). He invited Andy Mendelsohn up to the stage, and finally Mark Zuckman ran the live demos. The big deal? 11g. I attended several other sessions that talked about the new features in more detail, so there may be some repetition, but here is what I wrote down.

Configuration Pack, which in my limited understanding, is kind of like an implementation of Project Lockdown - allows and enforces higher security. To that end, the Audit Vault is currently in Beta; Audit Vault allows audit information to be gathered in a remote database. Along with that feature, there will also be Database Vault, which allows one to set up "realms" of data; supposedly, it could be setup such that not even a DBA could access the data in a realm. I can see headaches on the horizon with that one.

Flashback transactions, via Flashback Archive, was introduced. Alledgly, one may be able to go back *years*. Of course, you know the storage requirements for that one are going to be pricey. *grin* This feature has been dubbed "Total Recall". Simple command to enable at table level; alter table &table_name flashback archive;

Next up, Information Lifecycle Management (ILM). This feature is somewhat like automatic partitioning, allow some data to be automatically archived to older (slower) mediums. For instance, given a historical table, data that is highly critical can be kept on ultra-fast disk, slightly older data can be stored on legacy disks, and even older data can sit on commodity drives.

Compression technology used for RMAN backups can now be utilized against live DML, causing a very small footprint in overhead and saving 2/3 the amount of space.

Partitioning has been enhanced to allow reference partitions (ie, foreign keys), interval partitions (month-by-month), and finally a comprehensive composite partition that incorporates all of 'em.

SecureFiles: the next evolutionary step of LOBs; faster & better.

Automatic self-management: promises for faster performance and more automation. A popular slide (popular because I saw it in several presentations) shows how things improved from 9i to 10g to 11g in areas like Memory, IO, Storage, etc. I am taking it with a large grain of salt due to the massive problems we had with 10g Memory auto management.

The ILM features allow a true workload to be captured on a source database (being backported to 10g) and fully replayed on a target database.

And SQL Plan Change Control was announced, but no details.

For me, the most cool part was afterwards. I remained behind to plan my next move (which I missed because....). Ken Jacobs, Andy Mendelsohn, Mark Zuckman, someone important named Richard (not Niemiec, but with Oracle), Leng Leng Tan and several IOUG board members congregated right in front of me. I was able to eavesdrop in on their candid discussions, which was really cool. I even piped up a couple times so as to unashamedly included myself in some small way. I was shocked to discover that Mr. Jacobs had some issues with the Enterprise Manager GUI (which was being demoed live, as I mentioned). A couple others agreed; most of the complaints centered around the size of fonts, complexity of layout and navigation, and lack of eye candy. I was amazed. I wish I could have jumped in and said, "Yeah, MySql looks so much better!" But that was probably not the right place for me to say that. =) They also talked about all the auto features, and how they have heard some small rumblings about DBAs possibly being automated out of job. I was glad to have traded that time for the sessions I missed.




Monday, April 16th, 10:30
RAC SIG Expert Panel
Moderated by Rich Niemiec
Panelists: Barb Lundhild (Oracle RAC PM), Erik Peterson (Oracle RAC Liaison), George Trujillo, Prabhaker Gongloor (aka GP, Oracle Grid Control Expert)

For some reason, I did not take many notes for this session; I think many of the questions asked of the group were general in nature. I am hoping that someone else who attended got more out of it then I did, and will be posting their thoughts.

Anyway, I was very impressed that the RAC SIG has such strong support. Not only the above list, but also big-name players like Alex Gorbachev and Christo Kutrovsky from the Pythian group, and Dan Norris and Matt Topper from ITConvergence.
Erik Peterson seemed very charismatic and eagerly wanting the RAC SIG to thrive; he wants RAC to be perceived as a great tool, and is willing to go above and beyond to help others in whatever difficulties they may have. At least, that is my impression.

My personal take-aways from the Expert panel were:
- Automate as much as possible
- Use EM and stop writing custom scripts
- Make the CBO smarter (how?), and try avoid using hints
- Low maintenance is key!

There was also mention of the various websites and forums. The forums seem to be doing particularly well! =)
www.oracleracsig.org



Monday, April 16th, 2:15
Grid computing at Dell IT: Building the Megagrid
Logan McLeod
onmygrid.com

Logan has been the instigator of a 450 page/per month document, what he has affectionately labeled the "DBA Bible". Essentially, it is a comprehensive detail of all standards that apply to the DBAs job at Dell IT. Since Logan strongly believes in automation as the vehicle to adhere to standards, he feels that this massive encyclopedia is not only a "good thing" to have, but a requirement for all his DBAs. Logan also contends that automation removes the human-factor in errors, and allows the DBA to concentrate on more problem-solving and forecasting issues, opposed to the rote, dry hum-drum tasks of daily datatabase creations.

Enterprise Manager is highly advocated for monitoring and auditing standard compliance, especially in large environments with many databases.

Dell IT has implemented Shared Oracle Homes over NAS. I found this concept very intriguing and am wondering if it is feasible to apply this back home. It makes a lot of sense for particular scenarios; consolidates software trees, allows for a central area in which to do upgrades (or even downgrades, if one can retain older homes).

Dell IT has done a tremendous amount of research into various configurations with Linux and Oracle on Dell hardware. They have even gone to the extent of compiling a list of bug fixes that they feel are generally applicable for their validated configurations, allowing Dell IT to deliver recommendations to clients out of the box. As on Oracle Partner, they get their hands on the new toys, but more importantly, have a great working relationship with Oracle Support and developers, easing their research efforts.

Logan is a strong advocate of wrapping Oracle installs in RPMs, within Dell IT. I was not sure if he was recommending this for the general public or not, but it certainly seemed like a good idea if you have a package you want to deploy to RH boxes.



Monday, April 16th, 3:30
Virtualizing your development and test environments with VMWare
Matt Topper

I had high hopes for this presentation, and was very excited about VMWare. However, none of the demos worked which really took the wind out of the sail for me. I think Matt had some good things to share, and I still think VMWare has a lot going for it in the realm of test and development.



Tuesday, April 17th, 7:30
Breakfast with the board

Mike Abbey announce a few things. Introduced Peter Smith, a IOUG volunteer newly elected to the IOUG board, and Steve Lemme, newly departing. He also mentioned the Savvy Survey, and that some of the answers were going to be available during the session (one big reason why I attended). Some IOUG member benefits were also made mention of:
  • IOUG Best Practice Tip booklet
  • IOUG Discounts if one joins a RUG
  • OPEN (Oracle Professional Exchange Network)

I got to sit next to Peter, so that was kinda cool. Mike Abbey has a rough, sometimes dry sense of humor it seems. I talked to him after the breakfast and he came across really gruff. But I won that stinking survey, so I showed him. =) The survey was actually pretty interesting; covered several topics including facts about Las Vegas, Mandalay Bay, IOUG, Oracle and RUGs. I got a chance to quiz Ari Kaplan after the breakfast as well, and we were able to discuss other things, like how he is involved with some storage solutions at the U of I. I like Ari, he is friendly and easy to talk to. He is also a bit shorter then I expected, and almost soft-spoken.



Tuesday, April 17th, 9:45
ASM - A breakthrough in database storage management
Arulselvaraj Arogyanathan

I decided to attend this session at the last moment, as I was on my way to RAC Failover from the ITConvergence folks (Dan Norris and Matt Topper). I decided to do it because it had a lab, and I love labs. The funny thing is, they were having trouble getting the lab started due to problems with VMWare. What a coincidence; having just attended a session from Matt Topper the previous day *and* knowing that he was only a few doors away, I called him into the room. He had the problem fixed up in 5 minutes. Amazing!

On to ASM. I really enjoyed Arul's presentation. Ironically, I decided to not stay for the complete 2 hours, because there was another session I wanted to attend (next). Adding to the irony, some friends from CITES did stay, and they said the lab section sucked, because of all the problems they were having. The Lord moves in mysterious ways. *grin*

Arul did a great job at demystifying some things about ASM for the newbie. Or maybe it was simply one of those "Aha" moments for me, when things suddenly became a lot more clear. For starters, Arul showed how an ASM instance only requires a password file and a parameter file with two parameters (instance_name and instance_type). He made it very clear that there are no database files; no control files, no log files, no datafiles. He pointed out that one of the best practices (as pontificated by Arup Nanda) is to put ASM in its own Oracle Home (not shared), which allows the ASM to be patched independent of databases. There was some discussion of striping ASM over hardware striping. Arul has come to the conclusion that there will be 100% degradation due to dual-striping. However, Job Miller forwarded me a benchmark on ASM the week prior and it said that there is actually a slight performance *boost*. I am sure that I misread or misinterpreted something, so I have forwarded the white paper to Arul for his own determination. I need to go back myself and re-read the article to make sure I know what I am talking about.



Tuesday, April 17th, 11:am
Understanding Oracle RAC internals
Barb Lundhild, Product Manager

As I was preparing for IOUG, I read a few of the RAC papers. Many of them referenced Barb Lundhild's papers, so I thought I could not loose by attending one of her sessions. And I was right. =)

From talking with others, it seems she used a previous 10g presentation, and added some 11g features (which means she cannot post the presentation "as is").

For starters, 11g has enhanced the code for Split Brain conditions (when half or more of the nodes break off from the other half or less). Unfortunately, I did not take very good notes on this section for some reason. But 11g also adds the ability to dynamically increase the number of voting disks. Disk timeout is now a bit higher; I believe it used to be on the order of a second. I cannot read what I wrote, but disk timeout is also higher than another timeout (network? Something count), which means that the voting disk is the final arbitrator of split brain decisions (or node eviction).

Barb talked about a x$ view. I was not sure if it was new (and have been too lazy to check it out), but took down notes just in case. x$ksxpia has information on the interconnect information. In the context of bonded NICs, name_ksxpia would have the name of the network interface (whether bonded or not).

The LMS process responsible for reading/writing blocks to the interconnect. Oracle highly recommends a maximum of 2 cpus for each RAC instance (I believe 1 cpu can also be 1 core in the context, but not a Sun hyperthread). You need to guarantee that LMS will be given cpu time (otherwise known as real time). Another reason why folks give srong warnings about running VMWare for RAC, which does *not* guarantee real time.

When tuning the Grid, you want to make sure that you never see "GC CR block lost" (or any kind of lost block, for that matter). This event represents significant problems in the interconnect if it increases.

The topic of svrctl came up. I am really confused why Oracle decided so strongly to downplay svrctl in 8i and 9i, and then bring it back with a vengeance for the Grid. Are we also going to get a instctl or sidctl? =) Anyway, Barb emphasized that when you stop a database in svrctl, that will stop the instance *and* the service. However, starting the database does not necessarily do the same thing.

New with 10.2.0.3, if the VIP (Virtual/Private IP) goes away, only the listener and the service will die, *not* the instance! I guess before everything was biting the bullet. I am thinking that this "feature" ties into the voting disk being the final arbitrator, otherwise node eviction might get a little tricky with the instance running but no VIP.

I wrote down that "Application VIP" is new 10gR2. I am not sure if I meant R3, but since I never heard of it, I cannot be sure. Apparently, it registers resources to the clusterware.

She dropped some acronyms which are probably old hat for those who use RAC already (which would exclude me).
ONS is the Oracle Notification System.
FAN is Fast Application Notification.
ONS used by FAN to publish and subscribe.

Finally, one last tidbit about 11g; OCI session pools subscribe to Load Balancing for runtime stats.



Tuesady, April 17th, 12:15
RUG Business meeting

Peter Smith actually recommended this meeting to me, and while I found it interesting, I wish I would have spent my time elsewhere. I was introduced to some large User Groups like Rocky Mountaint and Utah, but personally I had absolutely no interest in growing my RUG. It was good to meet Tony Jedlinski and Megan Price; they had some good, practical things to share in a private conversation.




The rest of Tuesday
I made some poor decisions about which sessions to attend in the afternoon. I started with another 11g presentation (High Availability) by Ashish Ray. While Ashish is a good presenter, I found too much of the information to be review; he spent a lot of time on building the big picture from 9i to 10g. Probably spent less then 20 minutes total on 11g, and even that I had already heard in other sessions (ie, General Session, Barb Lundhild's RAC class). I wish I had done Alex Gorbachev's "Change Block Tracking inside out" as I initially planned; at least I have his paper. I next went to the RAC SIG Roundtable. This was a mixed bag for me; I felt way out of my league. =) For starters, I have zero experience with RAC. But it was cool to hobnob with the Pythian Group, Kyu-Mok Strickland from Boeing, Logan McLeod, Dan Norris, Erik Peterson, an intelligent fellow from GE Healthcare (Ryan? Brian?), and a couple other sharp folks whom I did not get names for. I kinda wish I had gone to Arup Nanda's "Performance Tuning in RAC", even if only to meet him and hear him speak. There were two other RAC sessions, but both were 2 hours long, and there is no way I had the mental stamina for a 2-hour session that late in the day.

UPDATE
I found my notes from Ashish's 11g HA presentation; don't know where they went.

ASM enhancements
  • automagically repairs corrupt blocks that are mirrored
  • fast resync of mirrored copy from transient errors (uses changed blocks)
  • rolling upgrades for ASM instances

Flashback: back out transaction and all subsequent conflicts
Data Archive: archive cannot be modified, data has a retention policy
Data Recovery Advisor: diagnosis based on symptoms
RMAN: intra-file parallelization; duplicate/clone/standby over-the-wire
Dataguard: read-only physical standby; logical standby is even more flexible
Snapshot Standby: Redo data still shipped, but not applied. Rolls back standby using flashback
Fast-Start Failover: supports ASYNC log transport (MAX PERFORMANCE)

At one point, Ashish was pressed for time so he summarized a few slides by saying "Everything is faster."

Streams
  • Source & Target compare & converge
  • Split/merge for Hubs
  • Cross-database LCR tracking

Online patching of one-off patches. Does not require RAC or dataguard.
Rolling upgrades via transent logical standby



Wednesday, April 18th, 8:30
Your 10g Tuning Arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj

One of the main reasons I attended this session was that I wanted to meet John, but I was also interested in his tuning techniques since he is well-known in this area. Unfortunately, I felt like I did not learn much from this presentation, mostly because I already use most of those tools and the session was more of an introduction than I was expecting. I am glad I met John; I ran into him several times after that and was able to talk with him.



Wednesday, April 18th, 9:45
Oracle Executive Session: Edward Screven, Chief Oracle Architect

Unfortunately, Mr. Screven could not make it, so two well-experienced Oracle specialists stood in for him; Tammy Bednar and Jack Brinson. Jack rubbed me the wrong way. =) Their presentation focused mostly on security features of 11g, which was good, but I was really hoping for some high-level pictures from Oracle's direction.

Take-away from this session:
- Database Vault is being backported all the way to 9.2.0.8; Oracle has never done this before
- Auditing best practices coming out soon

Most of the rest of the stuff had already been hinted at, or I simply was not interested in. I already had a picture of what Database Vault and Audit Vault were, and while the presentation had a couple more slides, there was not a whole lot of low-level detail.



Wednesday, April 18th, 11:am
Oracle Database 11g: Flashback
Daniel Liu (Oracle Principle Solution Architect)

Overall, I thought Daniel spent way too much time reviewing Flashback for 9i and 10g. I can understand that 11g builds on existing technology, but for a presentation that is meant to expose new features, we sure covered a lot of old territory. As a result, I did not come out of this session with much new information. I had already heard about Flashback Archive; the RVWR process was new to me. It works basically the same as LGWR, but writes the information in Reverse (hence the RV I believe).

Daniel also mentioned that it would be possible to flashback a standby database, but I was not able to ascertain exactly how that was done. As I learned from a different session, the standby databases in general have some nice features, and the ramifications of a Flashback Archive will be very interesting. For instance, 11g will make it rather easy to have a snapshot clone that you can write to, then flash it back to a point in time and re-synchronize with Production. Rinse and repeat.



Wednesday, April 18th, 12:15
Lunch and Learn
Wolfgang Breitling

I had already had a private meeting with Wolfgang on Sunday, so I did not feel like I *had* to go to the Lunch and Learn. But I thought it would be very interesting. Our conversation topics were not necessarily geared towards Wolfgang's specialty that much (10053 trace, stats, histograms and PeopleSoft), but I did get a chance to talk with John Kanagaraj and Kyu-Mok Stricklin (Boeing) a bit. I felt good about doing some networking. And we had some excellent things to share with each other; at least, I hope what I shared was on the good side of "ok". =)



Wednesday, April 18th, 1:30
Oracle Database 11g: Next Generation Self-management and Change Assurance
Leng Leng Tan (Oracle VP for Database Management)
Mughees Minhas (Oracle, but I did not get his title)

I am kicking myself for not introducing myself to Mughees; Job Miller has told me a bit about him and has passed on some papers that I respected and appreciated.

Leng Tan did the presentation, and I got a lot of out it. This was my best choice of the day! =)

ADDM for RAC - identifies global performance problems:
  • instance response time skew
  • cache fusion interconnect issues
  • lock manager issues
  • hot blocks
  • global high-load SQL

I am curious why RAC would not have these seemingly essential qualities all the way from 9i. It is rather odd that these great features are making a debut 2 versions later. Wow. At least they are finally coming out.

There was a lot of information for Automatic SQL tuning as well; pretty amazing stuff. They are finally moving down the path of having the database automatically tune all the sql that is being thrown at it. I say "finally" because they sold 10g as doing this, but the fine print spelled out that 10g really just lays the foundation to make automation possible. 11g actually implements it. What I consider to be a slightly downside, the automatic tuning seems to concentrate on SQL Profiles; while I think that is a considerable and effective solution, it is more like putting a band aid on the problem, while not fixing the root problem itself. If I recall correctly, the new tuning features will be analyzing all SQL statements out of the box, and can report on known issues (bad indexes, poorly written query, etc). Another aspect of this idea which I thought was a good decision is that even though the analytical part of the feature will happen all the time, the actual implementation part (ie, making the Profile "real") is purely optional. The tuned sql will be stored in AWR, and Robert Freeman mentioned that they will be retained for about a year. I find that hard to believe, but will be interested in seeing the real thing.

There are some new advancements for partitions; aside from having a truly comprehensive composite key, we are also getting a interval partition, along with functionality to automatically create new partitions based on that interval. Since we have a need for that exact feature now (and I am sure others as well; Tom Kyte mentioned that he uses rolling partitions based on date), I am really looking forward to this.

Database Replay; Wow, this is going to be big, I think. It will be a database option, and has the ability to capture a true workload into flatfiles. Yes, strange choice there, why flatfiles? Not sure about that. Again, this is building on one of the promises of 10g; when the introduced AWR with 10g, there was a lot of hype about capture workloads, but I never saw a real implementation of it. Now we can run the application and actually capture everything necessary to replay it back, verbatim, on any database. The replay part is an Oracle delivered client which reads the flat file; I was not able to extract more information about what that client actually was, but it will come from Oracle. It has features to match the timings from the recorded session, and does not need to be parallelized externally. I am still wanting more details, like how will it handle an underpowered target database, and how does it simulate data output (what do you do with 50 million rows returned to the screen?).

Along the same lines, there will be an option for SQL Performance Analyzer, which works very similarly to Replay, but replays everything linearly, with no concurrency. I am confused why this is needed if you are already using the Automatic SQL tuner; perhaps to benchmark queries that the auto-tuner cannot further tune? I will have to research that a little more.

Leng Tan threw in a couple slides about new provisioning features (available in 10gR3); I thought they were out of place, but I'll mention it like I wrote it down. Apparently, one can stage a "gold image", allowing for easy scaling (either up or down) in a RAC. I should have taken more notes on this feature.

Along with the Automagically tuned SQL, Oracle is finally giving us much better Plan Management. It is actually this feature that allows plans to be stored for a year or so in the AWR. Features support Plan Baseline and subsequent comparisons, which ties into the Auto Tuner; apparently the auto-tuner will test all new plans and actually verify they are better. Of course, we have all been bit by "better" low-cost plans that run slow than "worse" high-cost plants. I am curious if that will be addressed. I do not want the CBO to globally discard plans simply because the cost is higher.

There will be an Automatic Diagnostic Workflow, which will be responsible for automatically create an incident when you hit an ora-600 (or other bugs?), and will do a "first failure capture" of all relevant trace files, alert log, etc. I got the impression it was like Configuration Manager with RDA rolled in. Leng presented some interesting pictures, which I am sure will be making the rounds on the 'net sooner or later; the process flow seems very attractive.

Some new features in Patch Automation are also be making a debut in 10gR3; I have not yet tried it, but apparently there is support for rolling patches. In 11g, there are also proactive advisories from Metalink and complete support for online, one-off patches (including the Diagnostic ones). Leng explicitly pointed out that they have a long-term goal of making the CPU patches (yes, I hate the redundancy of terms) fully online.

Some enhancements in Enterprise Manager; one will be able to group targets, along with functionality to act at a group-level. One will also be able to establish hierarchies within groups, and can rollup hierarchies into "systems". It seems like there is great interest in managing *everything* about an enterprise-wide database installation. =)



Thursday, April 19th, 8:30
Case Study: Migration of ASM for RAC and non-RAC
Sumit Sengupta

I had high expectations for this presentation, but I guess they were too high. While Sumit seemed very knowledgeable, I found most of the material to be duplicated from other sessions. The number of sessions offered on Thursday were very limited, and not much I was interested in attending.



Thursday, April 19th, 9:45
The Answer to Free Memory, Swap, Oracle and Everything
Christo Kutrovsky

I am glad I attended at least one presentation by the Pythian group (have I mentioned them enough?). And anyone who starts off with a reference to Douglas Adam's has got to be cool anyway. *smile* Christo's explanation and observations about memory were very valuable well conceived. He painted a picture that painfully describes how expensive memory on disk is, and how much faster cpu registers are. This paved the path for HugePages, a page table scheme that significantly increases the size of the pages. By such a drastic increase in size, he showed a 512 times reduction in the number of page tables, and a huge improvement in memory allocation.

The one possible downside is that you have to allocate the memory up front when the OS starts. HugePages do not swap! They are pinned in physical memory. In retrospect, I am thinking to myself that anytime you pin something in memory and decrease the number visits to disk, you are going to see increases in performance unless, of course, your memory is too small. *grin*

He wrapped it up by talking about the benefits of using direct_io, and due to the bugs in Oracle releases 9.2.0.6 and lower, he highly recommends using 9.2.0.7 or higher. Not sure about the 10g code base, but I assume there is a similar story there.

Christo was also kind enough to share his blog: pythian.com/blogs/authors/kutrovksy. (NOTE: the blog he put on his presentation was sans-authors, I believe, but you get a 404 Page not found if you try that)



Thursday, April 19th, 11:am
Nothing. Nada. There were only 3 sessions to choose from, and only one looked even remotely interesting. I used the time to go over my notes and start my review of the week at IOUG.



Thursday, April 19th, 12:15
IOUG Closing Session - The great tool debate
Moderated by Mike Abbey

I was shocked and underwhelmed at how sparse and poorly attended the closing session was. My guess is that many folks either skipped the whole day or left early in the morning. A poor way to end such a prominent convention. Plus, the closing session was a bit weird, for a closing session. I have to admit, the one and only reason I stuck around was to find out the results of the Savvy Survey. Even that was a bit lackluster; was I one of three people that even attempted it? I and another guy (Frank Ponds) earned boasting privs at the end of it all, which included a backpack of little goodies (Oracle Wait Interface, PL/SQL Programming, 3 IOUG shirts, a mini usb mouse, a Oracle usb Hub, a IOUG pen and a CD of all the posted papers), and a fee waiver for a IOUG 2008 University session. Considering that the University sessions are a bit exorbitant (in price), that is nothing to sneeze at, but chances are I will not even be going next year. I kinda feel sorry for Mike Abbey. He totally gave me a bad first impression, but to have the closing session and Savvy Survey with so few numbers was a little disheartening.

But wait, what about the Great Tool Debate?!? Oh yeah, almost forgot that. =) I really like what George Trujillo emphasized again and again. No matter what tool you use, in the end it comes down to knowledge and experience, and today's typical shop has done a great disservice to application deployment in general by severely segregating the DBAs from the Developers. His comments got me to thinking that perhaps we should exercise a bit more cross-training. I come from a programming background, and have tried numerous times to get a foot in with the developers in their world. However, the one big bad dragon I stay away from is our ERP, a nasty beasty called Banner from SunGard/SCT. That should probably change.

Rich Niemiec has a court jester side to him; his personal presentation of himself is confident and authoritative (the black suite, black, slick-backed hair) but he often has something humorous to inject, sometimes delivered in a very dry, non-sense way that may catch you off guard. My favorite comment of his; ".Net (dot Net) is .dying (dot dying)". I was not familiar with the others, who represented APEX, PeopleSoft and Java shops. A number of people (most notably Rich) said Ruby on Rails is going to be the wave of the future next. For me, I don't really care; a tool is a tool, and as someone stated (George?), you have to use the right tool for the job, there is no silver-bullet here.


In conclusion
There were many sessions I wish I had attended. And I am really excited about going back and reading presentation notes from other folks who attended Collaborate07, not to mention trying to work my way through some of the posted presentations and papers. There is just so much! As this was my first major Oracle event, and only the second convention of my career, I feel that I could have done a better job preparing; mostly in the form of deciding for sure exactly which sessions I wanted to attend, and why. I had a list of "primary" and "secondary" choices, but that all went out the window as I reconsidered my options. For example, I completely did not factor in the fact that no 11g papers would be available, so if you wanted the information, you pretty much had to go to the session. Unless you have friends in high places. *grin* Additionally, one of my colleagues attended the convention, not to mention several friends from a sister department; I wish I could have collaborated with them a bit more, both before and during.

Tuesday, April 17, 2007

Collaborate07

I have been overwhelmed, impressed and even frustrated and disappointed with my first experiences at the annual IOUG event.

I will go into more detail when I can. But let me paint as picture of IOUG so far.

The convention is being held at Mandalay Bay Resort in Vegas. Mandalay Bay is HUGE! Not to mention that one can live, work and play without even stepping outside for a fresh of breathe air. Being Vegas, there is the obligatory casino 24x7, but outside that, there are a slew of high-end restaurants, vast open areas, classy shops and casinos. The convention itself is massive; we occupy roughly 3 floors, each with impressive 20 foot ceilings (or more in some cases). There are over 1000 sessions total, spanning IOUG, OAUG and Quest. IOUG has a mere 241 sessions. I have been trying to focus on RAC (including the RAC SIG, which I have somehow been able to avoid *grin*).

The first General Session is good. We heard from Ken Jacobs (Dr. DBA), Andrew Mendelhson and Mark Zuckman; they spoke about several things, and shared a few tidbits from 11g which look really good. Here are some of the highlights I took note of:
Audit Vault: Kinda like an audit repository
Database Vault: A protected region of database data that can be cordoned off from standard roles and privs, even "select any table" and such ilk.
Flashback Transactions: This sounded crazy, but interesting. You can conceivably flashback to any point in time spanning many YEARS. I have at least one outstanding question for that feature; "Where do you store all that data?"
Information Lifecycle Management (ILM): automates the migration of time-lapsed data (that which becomes less and less critical) to lower-end storage solutions (ie, DMX -> SATA -> Tape -> long term offline archive)
DML Compression: The same technology used to compressed RMAN backups now being leveraged against live update, delete and insert statements
Partitioning enhancements: Reference (foreign keys), interval (automatic range partitions), and complete composite partitions.
SecureFiles: The next step of evolution for LOB data. Makes me wondering what will happen to LOBs. Or what about those LONG datatypes still used in the 10g dictionary? =)
More automatic self-management: basically, all the automatic features become more automatic. *grin* I'll try to track down the diagram that depicts this really well.
SQL Plan Change Control: This sounded cool, but I did not catch all the details. More direct control over sql plans. Not sure if this will allow us to explicitly flush out a particular plan or not.

Another feature which I believed was mentioned in the context of ILM was the ability to capture a workload (even on 10g) and completely replay it (on 11g). I kinda thought 10g offered that out of the box with AWR, but never fully realized. Maybe it works now. *grin*

Exclusive
I happen to be sitting in front near the board of directors (I like sitting in front because my eyes are not the best and I hate glasses/contacts). When the session was over, several of the speakers and board memebers congregrated right in front of me, so I stuck around to enjoy their off-hand conversation. Wow, was I surprised. I would love to mention names, but in the sake of not being a weasel and especially not perpetuating rumors, I will just be vague. You can choose to ignore this if you like, or discredit it, or anything else.

To the point. In talking about the 11g features they just demo'd live with Enterprise Manager, the opinion was expressed that it would be nice if the GUI was a little more user-friendly, more graphical, nicer looking. I bring this up because I totally agree, and it seemed as if the folks were not merely venting to blow off steam, but making suggestions to another person who was present; call it constructive criticism. For some reason, I thought of the 10g Express Administrator (or was it MySql?); very simplistic and basic, but looks pretty. The Oracle Enterprise Manager is vast, huge, feature-rich, but also klunky and not always easy to navigate (my opinions, not those expressed, btw). Anyway, I thought it was really cool to catch a inside glimpse of "upper level" ruminations.

To that extent, I am now sitting in a ASM hands-on lab (before it begins, of course). I was going to attend Dan Norris's and Matt Topper's RAC class (down the hall), but I cannot pass up a lab class (well, I did yesterday, but .... that was yesterday). Anyway, the guys in the ASM lab were having trouble setting it up. Problem was VMWare. Dan Norris and Matt Topper just gave a presentation on VMWare, so I run down and get Matt. He comes in, and within 5 minutes has everything figured out. Wow, that was so cool. I played a small part in a larger play. And it felt good to network.

More later, that session is about to start.

Friday, March 23, 2007

RAC and getting involved

For the past couple weeks, we have been scrambling to figure out if we can implement RAC for a new, but ambitious, project. We have finally concluded that due to the deadline coming up so fast, we do not have enough in-house experience to push RAC as a fully-supported technology at this point in time.

However, that does not prevent us from reading books (K Gopal's "Real Application Clusters Handbook" is my current favorite), asking questions and essentially cramming our heads with information. Ironically, the hardest part has been acquiring the right hardware. A buddy gave me access to a VMWare installation, but getting the Cluster software installed and configured is a real pain in the but if you do not have console access. Must be other problems; one should be able to tunnel an x11 through ssh even for root, but I just have not gotten the right pieces in place yet. Our own department has been slow to provide hardware, not because the sysadmin do not want to, but because we are waiting to clear up the red tape. Grrr!

Regardless, I find the whole venture very exciting. I am glad we decided not to go with RAC, mostly because I feel it would make our DBA group look foolish. We have some really intelligent people on our team, and I believe in time we will have no problem supporting RAC (myself included). But to fast-track the whole thing and hope we will be ready is simply the wrong way to do a major project with national exposure.

In other events, I have started getting more involved with our local Oracle User Group a bit more. I am hoping that we can beef up the reputation of our OUG a little in several ways; brining in experts to talk about relevant and practical technologies (staying far away from product Sales!), present in-house white papers and provide an atmosphere of knowledge transfer and group learning in our technical community. Aside from our organization, we have some other high-tech companies around town, and I would love to bring the collective intelligence together.

But right now it is quite grass-roots. The current OUG president of our community started this thing about 2 years ago; even though it had a great start, I think interest has dwindled a little. Time to fuel the fire! =)

Wednesday, February 28, 2007

Setting up a logical standby database

Never having done this before, I wanted to give this a crack. Also, I was recently handed a project where we will either be using Streams or a Logical Standby. I was happy to learn that these two solutions are very similiar; one might even ask what the difference is. In the conventional sense, I think "Streams" implies a technology that is geared towards a set of objects, whereas logical standby is used for a whole database. At least, that is my impression early in the game.

To get started, I started reading the Oracle documentation about logical standby databases. This can be found in Chapter 4 of the Data Guard Concepts and Administration. Since I am using a really small and simple database (only non-system object is the demo SCOTT schema), I skipped the perquisite checks and stuff and jumped ahead to the part about setting up a Physical standby first. I got my primary all set up and used RMAN to make a physical standby. One can follow the directions from Chapter 3, but I have done a bit with RMAN duplication so I like it better, plus Paul Moen of Phythian has a great blog entry that details a simple recipe. In my example, I have the primary and the standby on the same host (remember, I said simple); for some reason, lock_name_space is deprecated in 10gR2, and I have been unable to find an alternative. So I use it anyway. One thing to keep in mind is that when using lock_name_space, you wind up with an instance_name that is not equal to your service_names (by default). TNSNAMES will have to point to the instance_name for the time being.

With a working, viable physical standby setup, I move on. The next couple steps are fairly straight-forward; stop the apply process in the standby and build the data dictionary in the primary redo logs. I am not concerned about role transitions (except transitioning the physical standby to logical, but the role does not change). The documentation makes a point about creating standby redo logs; I believe these are critical, but I still have a lot to learn about them. I created them, hence I also added a destination for local redo (as opposed to redo coming from the primary). What is disconcerting to me is the volume of redo that is generated on the standby; next time I will have to make them bigger (and possibly more of them, something to keep in mind when creating the controlfile with maxlogfiles).

The process of transitioning to a logical standby is actually pretty cool. Cool because it is so simple, yet the result is a bit complex. First, one has to convert the database:
ALTER DATABASE RECOVER TO LOGICAL STANDBY db_name;
This is where the lock_name_space and tnsname entry comes back to bite us (with the benefit of hindsight). If one is using a pfile (as I am), the in-memory name of the database will actually change, giving a warning about changing it in the pfile later on. I was impressed. However, now that the name has officially changed, make sure you remove lock_name_space, and that the name can be resolved properly by tns. Would be a good idea to actually pay attention to that warning and make the change in the pfile permament, as well. Another tricky thing about this operation is that it is expecting to read the data dictionary from the redo log. If it cannot find it, the operation will hang (forever).

After converting the standby, following the documentation, one has to shut it down to make some other changes, including the password file and a destination for local redo. With that, we are on the home stretch with the last couple commands:
ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;

Again, this is some slick bit of operation. As the documentation says, the standby is now essentially an equal copy, but "transactionally different". You have a fully functional database open for read-write, but still accepting redo from an external source. None of the messy overhead setting up a Streams database with various CAPTURE, PROPAGATE and APPLY handlers and various RULEs.

I did run into a bit of a problem right away, and still continue to struggle with this.

ARC1: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (1041)
ARC1: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
PING[ARC1]: Error 1041 when pinging standby STANDBY1.

oerr ora 1041
01041, 00000, "internal error. hostdef extension doesn't exist"
// *Cause: Pointer to hstdef extension in hstdef is null.
// *Action: Report as a bug


Nice error message, eh? Looking at Google, this error has been around in some form or fashion since Oracle 7. Yuck. For some reason, whenever my logical standby is bounced, the primary looses the handshake. I can resolve this by bouncing the primary, but of course this is far less than optimal, and I need to figure out what it is doing. Rather annoying. At first I thought it was because the listener was registering the standby instance fast enough, but after adding an entry in listener.ora, I was still getting this error.

I also played with creating a conflict on the standby by updating scott.emp in the standby, then the primary. The standy noticed the conflict right away and made a note about it, registering subsequent redo logs.

LOGSTDBY stmt: update "SCOTT"."EMP"
set
"SAL" = 810
where
"EMPNO" = 7369 and
"SAL" = 800 and
ROWID = 'AAAAAAAAEAAAAAyAAA'
LOGSTDBY status: ORA-01403: no data found
LOGSTDBY id: XID 0x000a.015.000005c1, hSCN 0x0000.000372b0, lSCN 0x0000.000372b0, Thread 1, RBA 0x00b8.0000002a.10, txnCscn 0x0000.000372b9, PID 18086, oracle@xxxxxx (P004)
LOGSTDBY Apply process P004 pid=25 OS id=18086 stopped
Wed Feb 28 11:08:39 2007
Errors in file /u01/app/oracle/admin/STANDBY1/bdump/standby1_lsp0_16624.trc:
ORA-12801: error signaled in parallel query server P004
ORA-01403: no data found
LOGSTDBY Apply process P006 pid=27 OS id=18095 stopped
LOGSTDBY Apply process P007 pid=28 OS id=18098 stopped
LOGSTDBY Apply process P005 pid=26 OS id=18092 stopped
LOGSTDBY Apply process P008 pid=29 OS id=18100 stopped
Wed Feb 28 11:08:43 2007
LOGMINER: Log Auto Delete - deleting: /u02/oradata/STANDBY1/log_STANDBY1_615653782_1_183.log
Deleted file /u02/oradata/STANDBY1/log_STANDBY1_615653782_1_183.log
LOGSTDBY Analyzer process P003 pid=24 OS id=18082 stopped
Wed Feb 28 11:10:15 2007
RFS[1]: Successfully opened standby log 3: '/u04/oradata/STANDBY1/redolog3STANDBY1.log'
Wed Feb 28 11:10:15 2007
RFS LogMiner: Client enabled and ready for notification
Wed Feb 28 11:10:15 2007
RFS LogMiner: Registered logfile [/u02/oradata/STANDBY1/log_STANDBY1_615653782_1_185.log] to LogMiner session id [1]


I still need to read about this registration process. Fearing that I would have "reinstatiate" the table (ie, like I might have to in Streams), all I did was update the table again back to what it was on the standby (no, I did not rollback because it was already committed; it was pretty much like a flashback), and restart the APPLY process. Very smooth and very intelligent.

Obviously, my early steps in Logical standbys and Streams in general are nothing new or earth-shattering for those who have already been around the block a few times. However, coming this far, I have to wonder why Streams is so difficult. For that matter, why are many other features so mind-boggling complicated and difficult to setup (LDAP or Advanced Replication just to name a couple)? Kudos to whoever worked on Streams, Logical Standbys in particular. While I still have some gripes about Streams in general, this exercise has given me some hope and a glimpse at a brighter future.

Sunday, January 28, 2007

Me and my job

Passing the OCM was a major accomplishment for me, a milestone in my career path. Hence my continued excitement; I have not had this strong of a sense of achievement... since... for a long time. *grin* And I truly believe it is "merely" a chapter in my career, a leg of the journey as I sojourn this enigma I call life. Preparing for and passing this difficult exam is not an end-all-be-all. Evidenced by the fact that there are many DBAs out there who are smarter, faster and more accurate than I am. The small fact that not many people have passed this exam is probably more of a testament to its secrecy and high price tag. Which is why I emphasize that the OCM is a stage, a rung of the ladder. With that image in mind, my end goal is to be really good! I read and converse with "Oracle Experts" like Tom Kyte, Jonathan Lewis, Dave Ensor and a large number of others whom I will not list out; if possible, I hope to claim that caliber at some point in my life. I call it "Striving for Excellence".

And here comes a helping of Humble Pie. Sure, just like any other human, I want to puff up my chest of pride, I want the world to recognize how good I am. Reality check - I am not so good that the world would even notice. If there were a DBA version of American Idol, I am sure all three judges would say something like "It was ok". "Not great, dog". "Good, but not good enough".

But I yearn and strive to be really good at what I do. This blog is cast in the context of my job, an Oracle DBA; yet, in the rest of my life I strive for excellence as well. Father and husband are at the forefront, but also in serving my community and being active in local government. One of the biggest reasons for this desire goes back to my belief that there is a God (and I am not He); all my gifts and strengths I credit to my God. Without His blessing, without His favor, I am nothing. Hence, it only makes sense to use what I have to bring Him honor, to make good on His investment. All too painfully I am often aware that my life is sometimes at odds with this view; I screw up. I, like you and everyone else, make mistakes. So while my goal is to be excellent, I have to be careful in depicting the fact that I am not there, yet. It is a dance, a choreographed movement with my heavenly Partner who leads my steps.

This is why I strive for excellence, to do what I can to make the big guy upstairs look good. And believe you me, I am thankful that He allows me to continue, despite my wretched performance from time to time. So I proudly add this label, this certification, to my somewhat short list. I am happy to be an Oracle Certified Master. Very happy!

Thursday, January 25, 2007

Oracle has deemed me fit to be Certified as Master

Wow, what an awesome feeling!! More later - today I am busy telling folks and planning for the future. =)

Thursday, January 18, 2007

A probing question

One of my OCM proctors (himself a 9i/10g OCM) poses this question to determine how well you understand Oracle architecture:
How does a transaction get processed in a Shared Server Installation?

The idea is that if you answer in a flat 5 minutes, you are screwed. =) One should first understand how a transaction is processed in a single-threaded (easy) environment, and then know the multi-threaded architecture well enough to incorporate knowledge of how a transaction is processed. He said that the answer should take about 30 minutes or so.

I bring this to light because I am making this one of my goals. I need goals to shoot for because I often find that I need a sense of direction. I am one of those people that, when mowing a lawn, cannot walk straight unless I set my eyes on my destination. Funny, that.