We had a long day today, so this will be significantly shorter than yesterday's update. The good news is that we had a ton of hands-on experience today; almost 80% pure labs and very little lecture. Just the way I like it. =) We installed and configured Clusterware, ASM, the Agent and the database software (but not the database, yet). Amazing stuff.
Yesterday's news about Grid Control R3 was extremely exciting, but when I got home I was disappointed. I specifically asked if he meant 10.2.0.3, and Andy specifically said "No, Release 3". Well, he referred us to oracle.com/enterprise_manager, and what do you see? 10.2.0.3. I installed 10.2.0.3 into our Production Grid Control 2 weeks ago. Oh well. At least it is encouraging to hear that the version we have installed is receiving so much good press. In fact, Andy is of the opinion that Release 3 is the only version worth getting. Not quite sure what that means.
Since we were heavy into labs today, I did not take many notes. Also, we are spending an extra hour at class these next couple days, not to mention that I am going in early tomorrow. Bottom line, short post so I can get to bed. *grin*
But here are some tidbits. This will probably be old hat for those who already have experience with ASM, and keeping in mind that we are working with Grid Control Release 2 (some of the things we did today are packaged much better in Release 3, or so I am told).
Order of installation:
- clusterware
- ASM
- Oracle Software
- Management Agent
- Database
It is possible to use sqlplus to shutdown all instances in a cluster, but unwise (use svrctl or GC instead). Shutting down each node individually is like an instance failure; read yesterday's post about remastering locks.
Since ASM only allows one datafile per tablespace, might as well use BIGFILEs.
As we were setting up user equilvalence (a fancy way of saying ssh key-pairs), I got to thinking "Why is this so complicated, so prone to error?" There is so much more that Oracle could do to automate the preparatory steps, even if only to provide a script to do it. Given all the automatic that Oracle Grid Control does (heck, you can provision a Gold Image unto bare metal!!), I am surprised these little things slip through the cracks sometimes.
RAC redo logs experience significantly more i/o than non-clustered databases. Andy recommends that the redo logs by physically isolated diskgroups. While you are at it, why not make them raw (since you are never going to back them up or recover them anyway, right?).
Back to Undo segments. Still on the thought that Undos are nothing more than Public Redo in a LMTS, Andy went to lenghts to demonstrate "how to eliminate Snapshot too old" errors. To boil it down and steal his thunder, the simple solution is to double the size of your UNDO tablespace. Or quadruple it. However, the longer answer involved more of Andy's famous pictures, and I learned a lot about Undo segments and how transactions use them. More later (hopefully).
As we were wrapping up, Andy talked about srvctl needs the name of the database as a parameter. This is because your OCR (Oracle Cluster Registry) is a solo act, there is only one; and the OCR has information for all the clustered databases on the node. However, I am thinking to myself, "Hmm... our development hosts have 40 or 50 databases, but sqlplus does not need the database name". Srvctl really came back with a vengeance; it is going to take us a while to get used to it. I think it is conceptually cool, but it adds an unwelcome layer of complexity.
Ciao
Tuesday, June 12, 2007
Monday, June 11, 2007
RAC Admin class, Day 1
Wow, for a first day, Andy sure packed a punch. I am up in Chicago taking the 10g Real Application Cluster Administration class with Andy Fortunak. Andy is great, and I refuse to take any more DBA classes from other instructors unless they are on my short list (Andy Fortunak, Sue Jang or Rick Pandya). I am not going to go over Andy's bio at this point, but he really knows his stuff and has been teaching for quite some time.
So, I was one of 12 folks in the classroom, each of us occupying one computer. We were given 3 class books (or rather, powerpoint printouts with embellished notes); two of the books are rather thick, the third is merely the last 3 appendixes (of 5). Actually, to put things in a little perspective, I learned a little later during the day that one of the fat books is 550 pages of lab material. Amazing! Full solutions with screenshots and step by step instructions. 550 pages!!
Andy is long-winded. Usually, this is a good thing, as he starts talking about the nitty-gritty details, stuff you will never hear from a sales person, a talking head or any Support Engineer. Probably not from any other instructor, either. But it took us about 30 minutes to get around to general introductions. The class is 5 days long, with 12 sections and the intro. Despite the books "recommended" curriculum, Andy plowed ahead with his own agenda. Typical Andy Fortunak. We spent the entire day (until well past the "normal" closing time of 5:pm) talking through the Introduction. And boy oh boy, what an intro.
First off, myth 1: RAC will make your system run faster.
Truth: RAC has the potential to make certain things run faster (especially in a warehouse), but more than likely, your application will run slower. Or maybe you will not notice any performance difference at all. It depends.
Services are big in RAC. Service Names are just the tip of the iceberg. A majority of 10g RAC Services are applications (ie, code, pl/sql, java, etc). While little used, Resource Manager is meant to be used with Services and can be quite powerful when wielded correctly.
Later on (later in the week), we will go over Backup & Recovery for the OCR file; it is that important.
Ignore Enqueue Services (GES), tune Cache Services (GCS).
A sidenote about Banner and RAC; BAD IDEA!! At least, until the fine folks at SGHE can rewrite the application to take advantage of parallelized operations. Otherwise, performance is going to suck and not worth the investment. If High Availability is a requirement, there are other alternatives with much better cost-benefit ratios. I cannot support the idea that the U of I be the ones on the cutting edge in terms of RAC. SunGard really needs to lead the charge in this, and they need to prove their leadership with large loads.
The Keep pool is important for RAC. More later.
RAC does not guarantee 100% uptime. If 1 node goes down, the database enters a brown-out time while the Lock Monitor (LMON) processes broadcasts to all surviving nodes that they must quiesce while the observing node remaster all global block locks. That is fancy-speak for saying that the a lot of work has to be done to redistribute the work that the now-dead node was doing. It takes a lot of time and resources to do that.
Automatic checkpointing (ie, fast_start_mttr_target = 0) in RAC is not good. You want better control over the checkpointing process because of all the extra events that trigger checkpoints and redo log buffer flushes in RAC.
Global Hash Partitioned Indexes can be good for non-partitioned tables with a sequential key.
Myth: UNDO is the next evolutionary step in Rollbacks.
Truth: UNDO = Public Rollback segments in a LMT (Locally Managed Tablespace)
RAC traditionally attempts to meet two objectives: scale-out and speedup. Scale-out is usually obvious; you add more hardware to handle more load. Speedup is more elusive; under optimal situations, you may see a max of %69 faster operations.
Andy drew many diagrams on the whiteboard which I will not replicate here. He went to lenghts to demonstrate how the degree of parallelism can quickly consume resources. This is mostly due to the fact that you get (2 * parallel degree) query slaves, each of which inherit specifications like Sort Area Size from the PGA. If your sort area size is 1Mb, and you set parallel degree at 4, you are not consuming a mere 1MB for sorting, but rather 9MB (1MB for the PGA + 8MB for the query slaves). Likewise, you also exponentially exacerbate the load on the CPU for each degree of parallelism. You start to get the picture that RAC is about parallelizing work. Tread here carefully. RAC solves memory and CPU bottlenecks by distributing the load among several nodes.
Global Temporary Tablespace (or Temporary Tablespace Groups) are also another big thing. RAC has the ability to automatically associate an affinity between a Temp Tablespace in a Group with a participating instance in the node if you have at least one "Global" Temporary Tablespace per node. Also, since Temp tablespaces never need recovery (recovery is impossible), Andy suggests that you never mirror the disks (Striping, ala RAID 0, is still a good thing, though).
Andy dropped a hint that Enterprise Manager R3 is out, and highly recommends using it over R2. Clarification, the 10.3.0.x version of Grid Control, not 10.2.0.3 of the database. He is supposed to provide a link for that in the near future.
Expect the SYSAUX tablespace to grow very quickly. This is because each instance maintains its own copy of the AWR, which is stored in SYSAUX. The default retention is 7 days; imagine what happens if you want to save 32 days.
Like Temp Tablespace Groups, Andy also recommends one UNDO tablespace per node. Alledgedly, there is a parameter that aligns each UNDO with a specific node.
As opposed to the well-known practice of multiplexing controlfiles 3 ways, Andy suggests that controlfiles be multiplexed 2 ways over mirrored disk (RAID 1, not unlike a REDO setup). This is mostly for recoverability purposes; RAC is going to hit the controlfile even harder.
Andy says:
Apparently, Oracle is focusing more on the controlfile to keep track of RMAN. However, the one exception is that the catalog can still be very useful with Data Guard, due to the fact that the standby controlfiles are different than the primary's.
RAC Redo logs can be configured differently for each instance. I have yet to find out why (why it is possible, and why you would want to in the first place).
Andy was very concerned about the interconnects, Cache Fusion and Block Transfer. He is of the opinion that you need to spend the big bucks on the interconnect hardware. Interesting how Dell did not have quite the same opinion. It will be interesting to see how things bear out for us, but I suspect that with our small load, we will be fine with the "commodity" Gigabit NICs for the meantime.
Andy was also quite adamant about not using the Flashback Logs. I think he will follow that up with more details later in the week.
We spent a good amount of time on Lock Manager. At startup, RAC propagates a complete list of all blocks and assigns a lock for each one (yes, every single one) to each participating node. Each lock starts with NULL, and there is quite an interesting scheme to upgrade and downgrade block lock levels. What it comes down to is that downgrading a lock take a long time. Andy went to pains to demonstrate this with an extremely detailed picture (he is known for such pictures). Also typical of Andy, he provided a number of "extra" slides that he had developed over the years (some dating back to 8i OPS), and gave some good reference information for the various lock states. Essentially, it is a very expensive operation to downgrade a lock because the node that masters the lock must flush the redo log buffer. So if Node 2 requests a block lock for a block that is already in Exclusive mode on Node 1, Node 1 must first downgrade the lock causing the log buffer to be flushed, then upgrade the lock mode for Node 2, and finally transmit the block and associated Undo information (2 additional blocks) to Node 2. You can see why an OLTP system would have heyday with that. And as mentioned earlier, when a node crashes, all those mastered block locks must be redistributed amongst the surviving nodes.
And that was only what I wrote down. There is a lot more on the whiteboard, in the class book and in the supplemental slides.
Tomorrow we will dive into a full day of labs. Andy has promised that it will be challenging and frought with sandtraps; if we make a mistake, it takes about 7 hours to rebuild the computer. Don't ask me why; I have always been amazed at how disruptive and slow these computer rebuilds are for Oracle classrooms. But back to tomorrow. We are paired up such that each pair of computers will become one node in a two-node cluster. One person will be Node 1, who, it is rumored, will do more work. Node 2 must be done in a synchronized, serial fashion. I guess we will learn more. My partner is Bob, a Sys Admin from Northern Trust. I figured between the two of us, we have all the OS and DBA skills covered. =)
The rest of the week promises to be grueling. We will be starting at 9 and marching all the way to 6 (at the earliest). Hopefully we will have some reprieve on Friday. I am excited about the lab section tomorrow; we will be doing some raw install and configurations, starting with naked hardware. I hope to pass this experience on to my colleagues who are eagerly awaiting hardware of our own.
So, I was one of 12 folks in the classroom, each of us occupying one computer. We were given 3 class books (or rather, powerpoint printouts with embellished notes); two of the books are rather thick, the third is merely the last 3 appendixes (of 5). Actually, to put things in a little perspective, I learned a little later during the day that one of the fat books is 550 pages of lab material. Amazing! Full solutions with screenshots and step by step instructions. 550 pages!!
Andy is long-winded. Usually, this is a good thing, as he starts talking about the nitty-gritty details, stuff you will never hear from a sales person, a talking head or any Support Engineer. Probably not from any other instructor, either. But it took us about 30 minutes to get around to general introductions. The class is 5 days long, with 12 sections and the intro. Despite the books "recommended" curriculum, Andy plowed ahead with his own agenda. Typical Andy Fortunak. We spent the entire day (until well past the "normal" closing time of 5:pm) talking through the Introduction. And boy oh boy, what an intro.
First off, myth 1: RAC will make your system run faster.
Truth: RAC has the potential to make certain things run faster (especially in a warehouse), but more than likely, your application will run slower. Or maybe you will not notice any performance difference at all. It depends.
Services are big in RAC. Service Names are just the tip of the iceberg. A majority of 10g RAC Services are applications (ie, code, pl/sql, java, etc). While little used, Resource Manager is meant to be used with Services and can be quite powerful when wielded correctly.
Later on (later in the week), we will go over Backup & Recovery for the OCR file; it is that important.
Ignore Enqueue Services (GES), tune Cache Services (GCS).
A sidenote about Banner and RAC; BAD IDEA!! At least, until the fine folks at SGHE can rewrite the application to take advantage of parallelized operations. Otherwise, performance is going to suck and not worth the investment. If High Availability is a requirement, there are other alternatives with much better cost-benefit ratios. I cannot support the idea that the U of I be the ones on the cutting edge in terms of RAC. SunGard really needs to lead the charge in this, and they need to prove their leadership with large loads.
The Keep pool is important for RAC. More later.
RAC does not guarantee 100% uptime. If 1 node goes down, the database enters a brown-out time while the Lock Monitor (LMON) processes broadcasts to all surviving nodes that they must quiesce while the observing node remaster all global block locks. That is fancy-speak for saying that the a lot of work has to be done to redistribute the work that the now-dead node was doing. It takes a lot of time and resources to do that.
Automatic checkpointing (ie, fast_start_mttr_target = 0) in RAC is not good. You want better control over the checkpointing process because of all the extra events that trigger checkpoints and redo log buffer flushes in RAC.
Global Hash Partitioned Indexes can be good for non-partitioned tables with a sequential key.
Myth: UNDO is the next evolutionary step in Rollbacks.
Truth: UNDO = Public Rollback segments in a LMT (Locally Managed Tablespace)
RAC traditionally attempts to meet two objectives: scale-out and speedup. Scale-out is usually obvious; you add more hardware to handle more load. Speedup is more elusive; under optimal situations, you may see a max of %69 faster operations.
Andy drew many diagrams on the whiteboard which I will not replicate here. He went to lenghts to demonstrate how the degree of parallelism can quickly consume resources. This is mostly due to the fact that you get (2 * parallel degree) query slaves, each of which inherit specifications like Sort Area Size from the PGA. If your sort area size is 1Mb, and you set parallel degree at 4, you are not consuming a mere 1MB for sorting, but rather 9MB (1MB for the PGA + 8MB for the query slaves). Likewise, you also exponentially exacerbate the load on the CPU for each degree of parallelism. You start to get the picture that RAC is about parallelizing work. Tread here carefully. RAC solves memory and CPU bottlenecks by distributing the load among several nodes.
Global Temporary Tablespace (or Temporary Tablespace Groups) are also another big thing. RAC has the ability to automatically associate an affinity between a Temp Tablespace in a Group with a participating instance in the node if you have at least one "Global" Temporary Tablespace per node. Also, since Temp tablespaces never need recovery (recovery is impossible), Andy suggests that you never mirror the disks (Striping, ala RAID 0, is still a good thing, though).
Andy dropped a hint that Enterprise Manager R3 is out, and highly recommends using it over R2. Clarification, the 10.3.0.x version of Grid Control, not 10.2.0.3 of the database. He is supposed to provide a link for that in the near future.
Expect the SYSAUX tablespace to grow very quickly. This is because each instance maintains its own copy of the AWR, which is stored in SYSAUX. The default retention is 7 days; imagine what happens if you want to save 32 days.
Like Temp Tablespace Groups, Andy also recommends one UNDO tablespace per node. Alledgedly, there is a parameter that aligns each UNDO with a specific node.
As opposed to the well-known practice of multiplexing controlfiles 3 ways, Andy suggests that controlfiles be multiplexed 2 ways over mirrored disk (RAID 1, not unlike a REDO setup). This is mostly for recoverability purposes; RAC is going to hit the controlfile even harder.
Andy says:
Oracle is moving away from Recovery Catalog like nobody's business
Apparently, Oracle is focusing more on the controlfile to keep track of RMAN. However, the one exception is that the catalog can still be very useful with Data Guard, due to the fact that the standby controlfiles are different than the primary's.
RAC Redo logs can be configured differently for each instance. I have yet to find out why (why it is possible, and why you would want to in the first place).
Andy was very concerned about the interconnects, Cache Fusion and Block Transfer. He is of the opinion that you need to spend the big bucks on the interconnect hardware. Interesting how Dell did not have quite the same opinion. It will be interesting to see how things bear out for us, but I suspect that with our small load, we will be fine with the "commodity" Gigabit NICs for the meantime.
Andy was also quite adamant about not using the Flashback Logs. I think he will follow that up with more details later in the week.
We spent a good amount of time on Lock Manager. At startup, RAC propagates a complete list of all blocks and assigns a lock for each one (yes, every single one) to each participating node. Each lock starts with NULL, and there is quite an interesting scheme to upgrade and downgrade block lock levels. What it comes down to is that downgrading a lock take a long time. Andy went to pains to demonstrate this with an extremely detailed picture (he is known for such pictures). Also typical of Andy, he provided a number of "extra" slides that he had developed over the years (some dating back to 8i OPS), and gave some good reference information for the various lock states. Essentially, it is a very expensive operation to downgrade a lock because the node that masters the lock must flush the redo log buffer. So if Node 2 requests a block lock for a block that is already in Exclusive mode on Node 1, Node 1 must first downgrade the lock causing the log buffer to be flushed, then upgrade the lock mode for Node 2, and finally transmit the block and associated Undo information (2 additional blocks) to Node 2. You can see why an OLTP system would have heyday with that. And as mentioned earlier, when a node crashes, all those mastered block locks must be redistributed amongst the surviving nodes.
And that was only what I wrote down. There is a lot more on the whiteboard, in the class book and in the supplemental slides.
Tomorrow we will dive into a full day of labs. Andy has promised that it will be challenging and frought with sandtraps; if we make a mistake, it takes about 7 hours to rebuild the computer. Don't ask me why; I have always been amazed at how disruptive and slow these computer rebuilds are for Oracle classrooms. But back to tomorrow. We are paired up such that each pair of computers will become one node in a two-node cluster. One person will be Node 1, who, it is rumored, will do more work. Node 2 must be done in a synchronized, serial fashion. I guess we will learn more. My partner is Bob, a Sys Admin from Northern Trust. I figured between the two of us, we have all the OS and DBA skills covered. =)
The rest of the week promises to be grueling. We will be starting at 9 and marching all the way to 6 (at the earliest). Hopefully we will have some reprieve on Friday. I am excited about the lab section tomorrow; we will be doing some raw install and configurations, starting with naked hardware. I hope to pass this experience on to my colleagues who are eagerly awaiting hardware of our own.
Tuesday, May 22, 2007
Before I forget
We have been dealing with a number of issues here; a whole slew of number one priorities. =) So quickly, aside from Global Campus and all that fun, here are some other things we need to dig into:
1) A solid method (I hate when people say "methodology" when they could simply say "method") to apply Oracle Critical Security Patches (aka, CPU). Just came across the Best Practice White Paper (March 2007). One of the little gems I was alerted to (thanks Job!):
2) Clean up the state of our Backup & Recovery policies. I am tempted to say we need to rewrite the whole thing, but I do not think it is that drastic. We probably should look into incremental backups more closely (see previous post), get more aggressive with rman-to-tape backups in light of ASM, and make sure we are all good to go on all types of Recovery.
3) The Grid cometh..... Get more comfortable with Enterprise Manager Grid Control. I referenced Logan McLeod earlier; I think he gave us a little present in terms of documentation. Maybe. But I can not talk about it.
1) A solid method (I hate when people say "methodology" when they could simply say "method") to apply Oracle Critical Security Patches (aka, CPU). Just came across the Best Practice White Paper (March 2007). One of the little gems I was alerted to (thanks Job!):
You can detect potential patch conflict in advance by running the following command:
opatch apply -silent -no_bug_superset -report
The –report option detects any conflicts without applying patch.
2) Clean up the state of our Backup & Recovery policies. I am tempted to say we need to rewrite the whole thing, but I do not think it is that drastic. We probably should look into incremental backups more closely (see previous post), get more aggressive with rman-to-tape backups in light of ASM, and make sure we are all good to go on all types of Recovery.
3) The Grid cometh..... Get more comfortable with Enterprise Manager Grid Control. I referenced Logan McLeod earlier; I think he gave us a little present in terms of documentation. Maybe. But I can not talk about it.
Monday, May 21, 2007
Onsite with Dell
Last week (May 16th), we met with a number of technies at Dell. Overall, I think it was a good experience, and I am glad that we as a group were able to interact with some folks "in the know". While I think it was good that we went, there were some downsides that overshadowed our time.
The Dell PR folks did a great job of making a wonderful first impression and reaching out to us. Kudos to Rooney Thomas; I personally thought the organization and planning that went into our itinerary was excellent and was by far the best I have been treated. I am not merely talking about a most excellent dinner at Eddie B's (you gotta check that place out!!) and steep discounts at a super-friendly, classy and very comfortable Marriot. I am also talking about the networking that Rooney did to pull so many folks together for a little chat. Unfortunately, some of those folks were... how do I say this politely? Not exactly needed. But I was impressed with Rooney and the agenda he set up. Along with him were Craig (last name withheld because I simply forgot) and Carey Dietert (more techy/PR guys). I liked Carey - I felt he has a great grasp on the Oracle/Dell relationship, understands the technology really well and knows how to more forward. On top of that, he is a good communicator. There was actually an Oracle representative, but he came late, didn't say anything and left early. He handed out his business card, which I kept for the sole reason of mentioning in this blog and am now tossing away. Apparently he is some "Business Development Manager" for the Dell/Oracle Partnership. Into the bit bucket that one goes.
I will start with the bad and go from there. Rooney pulled together some senior folks in the IT department and about half of them could have stayed in their cubicles (or office or whatever). Take Peter (last name withheld to avoid flash mobs, death threats, etc). Even though he has a couple certifications (Microsoft being one of them *grin*), he is a "Development Engineer" and works with a number of Oracle technologies. Problem is, he was just plain wrong on a couple accounts. Other times, I had no idea why he was even talking. He liked to listen with his eyes closed and his arms crossed. Know the type? Now there are two cards adorning a little basket under my desk.
Another was a DBA-type. I did not get his name, but he presented something early on. Powerpoint and a talking head.... not my style. Fortunately, a lot of questions were asked during his presentation which helped to keep things interesting. Sometimes. A lot of tangents.
Aaron Burns showed up representing the PeopleSoft, and now SunGard BANNER, side of their collaborative testing efforts. He made a relatively (keyword, relatively) impressive pitch, and I have an email in to him. Still waiting to hear back, but I would love to know more. My boss says he saw the presentation at the SCT conference (SUMMIT) and discovered there were a lot of "smoke and mirrors". I am holding out hope that there is actually something worth mining in this contact.
So with all that, I have painted a picture that makes it seem like our trip was a complete waste of time. Well, for some of the Dell folks, I think it was. I got the impression that there was a common misconception that we were possibly considering a migration from our current E10k platform running Banner which is at the core of the University of Illinois to a Dell/Linux/RAC solution. Yes, I am sure they salivated over that. Imagine the almost audible bubble-popping sound when we made it very clear that we were simply considering a 2-node deployment for a infantile yet highly visible and somewhat risky online classroom concept.
In my opinion, the meat and potates were covered by the other folks that came out to play. I will fill in their names when I get them. One dude walked us through their operational centers, exhibiting rows upon rows of rack-mounted hardwired all to various kvm switches. In another room, rows up on rows of desks where employees were connecting to various numbers of the servers and testing things: RAC, Microsfot, PeopleSoft, etc. He gave us a live-demo of killing one node of a two-node RAC which seemed pretty canned to me. I think it was good for the others to see, as I saw something very similar at IOUG. A very simple test in a very controlled environment, yet practical for all that. We were able to ask a lot of relevant questions, talk to the engineers doing the work (those that were with us) and learn a lot just from the adhoc discussion.
For the last couple hours, we crammed into a room with a an offering of sandwiches and sides, engaging in more "Question and Answer". Another Dell Engineer joined us, and he seemed to be very knowledgeable, able to answer technical questions. There we hashed out ideas and a rough road map of how to proceed. Dell has been down the RAC road so many times they now have an image you can download into a CD, which can be used for a local "Big Bang" to jumpstart a typical install. They also have copious documentation, some of which they are willing to share, and a butt-load of advice. Logan McLeod joined us near the end and clarified a few things for us, but also painted a very realistic picture; a number of big shops are already running RAC, and more are heading that way. I have said this before, but he really likes to automate things, and in the context of what he said, I am a new convert. Automate the procedures to deploy a new databsae or new Oracle software. Automate and consolidate monitoring. And the very first step is to establish standards, which drive and focus the automation.
What did we come away with? I think some definites are the use of ASM and 10.2.0.3. Logan practically praised the efforts being done with 11g and is rather confident that 11g will be an unusual first release in that it will not be as buggy as most of Oracle RDBMS first releases. Obviouisly, we will let others test the water a little bit, but I think 10.2.0.3 is the way to go for now. Also, there is some interest to use Dell's cookie-cutter install, even if only to see how it tastes. We also came away with a better idea how to carve up the disk, and more realistic expectations; Linux kernel patches still require database-wide downtime, even with RAC.
I hope to add more later. We have a meeting tomorrow and I look forward to hear what the others have to say.
In closing, I was really impressed with Dell. They have some really smart people doing really smart things. They are confidently on the edge of Oracle's technology, and they have established their confidence in rigorous testing. Dell is aggresively going after the market niche of customers who want to stop paying exorbitant license fees for colossal hardware. Granted, RAC is not necessarily the "best of breed" for everyone; I think too often the technology drives the SLA, instead of the SLA driving the technology. It will be interesting to see how this relationship matures.
The Dell PR folks did a great job of making a wonderful first impression and reaching out to us. Kudos to Rooney Thomas; I personally thought the organization and planning that went into our itinerary was excellent and was by far the best I have been treated. I am not merely talking about a most excellent dinner at Eddie B's (you gotta check that place out!!) and steep discounts at a super-friendly, classy and very comfortable Marriot. I am also talking about the networking that Rooney did to pull so many folks together for a little chat. Unfortunately, some of those folks were... how do I say this politely? Not exactly needed. But I was impressed with Rooney and the agenda he set up. Along with him were Craig (last name withheld because I simply forgot) and Carey Dietert (more techy/PR guys). I liked Carey - I felt he has a great grasp on the Oracle/Dell relationship, understands the technology really well and knows how to more forward. On top of that, he is a good communicator. There was actually an Oracle representative, but he came late, didn't say anything and left early. He handed out his business card, which I kept for the sole reason of mentioning in this blog and am now tossing away. Apparently he is some "Business Development Manager" for the Dell/Oracle Partnership. Into the bit bucket that one goes.
I will start with the bad and go from there. Rooney pulled together some senior folks in the IT department and about half of them could have stayed in their cubicles (or office or whatever). Take Peter (last name withheld to avoid flash mobs, death threats, etc). Even though he has a couple certifications (Microsoft being one of them *grin*), he is a "Development Engineer" and works with a number of Oracle technologies. Problem is, he was just plain wrong on a couple accounts. Other times, I had no idea why he was even talking. He liked to listen with his eyes closed and his arms crossed. Know the type? Now there are two cards adorning a little basket under my desk.
Another was a DBA-type. I did not get his name, but he presented something early on. Powerpoint and a talking head.... not my style. Fortunately, a lot of questions were asked during his presentation which helped to keep things interesting. Sometimes. A lot of tangents.
Aaron Burns showed up representing the PeopleSoft, and now SunGard BANNER, side of their collaborative testing efforts. He made a relatively (keyword, relatively) impressive pitch, and I have an email in to him. Still waiting to hear back, but I would love to know more. My boss says he saw the presentation at the SCT conference (SUMMIT) and discovered there were a lot of "smoke and mirrors". I am holding out hope that there is actually something worth mining in this contact.
So with all that, I have painted a picture that makes it seem like our trip was a complete waste of time. Well, for some of the Dell folks, I think it was. I got the impression that there was a common misconception that we were possibly considering a migration from our current E10k platform running Banner which is at the core of the University of Illinois to a Dell/Linux/RAC solution. Yes, I am sure they salivated over that. Imagine the almost audible bubble-popping sound when we made it very clear that we were simply considering a 2-node deployment for a infantile yet highly visible and somewhat risky online classroom concept.
In my opinion, the meat and potates were covered by the other folks that came out to play. I will fill in their names when I get them. One dude walked us through their operational centers, exhibiting rows upon rows of rack-mounted hardwired all to various kvm switches. In another room, rows up on rows of desks where employees were connecting to various numbers of the servers and testing things: RAC, Microsfot, PeopleSoft, etc. He gave us a live-demo of killing one node of a two-node RAC which seemed pretty canned to me. I think it was good for the others to see, as I saw something very similar at IOUG. A very simple test in a very controlled environment, yet practical for all that. We were able to ask a lot of relevant questions, talk to the engineers doing the work (those that were with us) and learn a lot just from the adhoc discussion.
For the last couple hours, we crammed into a room with a an offering of sandwiches and sides, engaging in more "Question and Answer". Another Dell Engineer joined us, and he seemed to be very knowledgeable, able to answer technical questions. There we hashed out ideas and a rough road map of how to proceed. Dell has been down the RAC road so many times they now have an image you can download into a CD, which can be used for a local "Big Bang" to jumpstart a typical install. They also have copious documentation, some of which they are willing to share, and a butt-load of advice. Logan McLeod joined us near the end and clarified a few things for us, but also painted a very realistic picture; a number of big shops are already running RAC, and more are heading that way. I have said this before, but he really likes to automate things, and in the context of what he said, I am a new convert. Automate the procedures to deploy a new databsae or new Oracle software. Automate and consolidate monitoring. And the very first step is to establish standards, which drive and focus the automation.
What did we come away with? I think some definites are the use of ASM and 10.2.0.3. Logan practically praised the efforts being done with 11g and is rather confident that 11g will be an unusual first release in that it will not be as buggy as most of Oracle RDBMS first releases. Obviouisly, we will let others test the water a little bit, but I think 10.2.0.3 is the way to go for now. Also, there is some interest to use Dell's cookie-cutter install, even if only to see how it tastes. We also came away with a better idea how to carve up the disk, and more realistic expectations; Linux kernel patches still require database-wide downtime, even with RAC.
I hope to add more later. We have a meeting tomorrow and I look forward to hear what the others have to say.
In closing, I was really impressed with Dell. They have some really smart people doing really smart things. They are confidently on the edge of Oracle's technology, and they have established their confidence in rigorous testing. Dell is aggresively going after the market niche of customers who want to stop paying exorbitant license fees for colossal hardware. Granted, RAC is not necessarily the "best of breed" for everyone; I think too often the technology drives the SLA, instead of the SLA driving the technology. It will be interesting to see how this relationship matures.
Thursday, May 10, 2007
service names; an old feature that has matured
Reading through Jeremy Paul Schneider's Oracle Services paper (not yet publicly available). He talks a bit about how Oracle Services has grown over the years, and the impact of services in today's 10g databases. I am convicted, caught red-handed, because I was largely clueless about the important of service names. Thankfully, we switched most of our connection strings over to service_names in the past due to peculiar connection issues, but I see that we still have a few connection strings using SID. Need to change those.
I like Jeremy's paper; good humor, easy style, combined with intelligence and well-researched ideas. One of the drawbacks to reading good papers is that it makes me feel artificially smart. *grin* I am the type of person that learns slowly from books and usually needs to reinforce everything by getting my hands dirty.
I like the references as well. Quotes from Oracle Documentation, himself, his colleague Dan Norris (who sits 20ft from him), not to mention other experts in the field like S. Kumar, J. Morle and HOTSOS (Cary?).
UPDATE:
Jeremy posted the paper on his blog:
http://www.ardentperf.com/pub/services-schneider07.pdf
I like Jeremy's paper; good humor, easy style, combined with intelligence and well-researched ideas. One of the drawbacks to reading good papers is that it makes me feel artificially smart. *grin* I am the type of person that learns slowly from books and usually needs to reinforce everything by getting my hands dirty.
I like the references as well. Quotes from Oracle Documentation, himself, his colleague Dan Norris (who sits 20ft from him), not to mention other experts in the field like S. Kumar, J. Morle and HOTSOS (Cary?).
UPDATE:
Jeremy posted the paper on his blog:
http://www.ardentperf.com/pub/services-schneider07.pdf
Wednesday, May 09, 2007
Reading the documentation: incremental backups
Some smart people have recommended that I simply read the documentation from front to back. Tom Kyte suggested that when he was in town for our local IOUG, and Job Miller has mentioned it a few times. Reading is hard, especially a technical document. Even if you only focus on a few of the books; my brain is well-trained to expect sci-fi/fantasy when I read. *grin*
There is a ton of good stuff in the documentation. I have to give a standing ovation for the folks that write them. Yes, sometimes there are errors, or an old section is apparently skipped over during a version upgrade. But by and large, it is good stuff!
Just today I was reading about incremental backups. And I came across this little feature I had not realized before:
source
Wow, that is quite handy. So if you are ever tempted to put a DEV database in archivelog mode just to get point-in-time recovery, you can go with incremental backups instead. How cool is that?
Of course, there are a lot of other handy ramifications of using incremental backups (ie, a precurssor to snapshot standby's, you can convert your physical standby to a reporting database and then convert it back at the end of the day).
There is a ton of good stuff in the documentation. I have to give a standing ovation for the folks that write them. Yes, sometimes there are errors, or an old section is apparently skipped over during a version upgrade. But by and large, it is good stuff!
Just today I was reading about incremental backups. And I came across this little feature I had not realized before:
To reduce backup sizes for NOARCHIVELOG databases. Instead of making a whole database backup every time, you can make incremental backups.source
Wow, that is quite handy. So if you are ever tempted to put a DEV database in archivelog mode just to get point-in-time recovery, you can go with incremental backups instead. How cool is that?
Of course, there are a lot of other handy ramifications of using incremental backups (ie, a precurssor to snapshot standby's, you can convert your physical standby to a reporting database and then convert it back at the end of the day).
Tackling the 10g dataguard GUI
After hearing all these great things about the dataguard management in 10g EMGC (our shorthand for Enterprise Manager Grid Control) and having been badly scarred from the 9i dataguard implementation, I just had to try this out.
For starters, I tried to create a database. Strangely, EMGC which can clone Oracle Homes, deep-drill into performance problems, manage tablespaces, backups, workloads, stats, coffee cups, user objects, and can slice-n-dice the universe, this EMGC... I could not find a button to create a database. Maybe I missed it. Which is easy to do in EMGC, because it has so many features (most of which work quite well), so many links and buttons and gizmos and ways to get to where you think you want to go. But I could not find "Create Database". So I opted to clone instead. I was not disappointed, either. The prompts and progression of screens was very intelligent, logical and reassuring. I have found that to be the case with a number of "advanced features" in EMGC; of course, there are times when you come to expect such great performance and useability and you run head-first into a brick wall of errors. But I'll get to that in a second...
Database, check. Navigate to the Data Guard section under "Management", and the feeling that you are being well cared for continues. Create Standby. Oops, no spfile. EMGC reports that you need to create an spfile. Strange that the GUI does not offer to do that for you. It is really simple; "create spfile from pfile;" and reboot the database. Ok, spfile, check. Create Standby.
What does that mean? No details, no message, no mention of a log file. This is my most frequent and biggest complaint with EMGC to date. They often forget to inform the user where they can look for more details. "Go to the log file" you would think. This is EMGC, which of the many logfiles do you want to look in? I do some looking around and spot emoms.log, which has a couple more characters, but still quite useless:
So, gritting my teeth, I file an SR. I should have a separate blog entry on that. Filing SRs for OMS is not that fun. But... let us not digress anymore.
So after 20 hours, my SR is still unsolved. Fortunately, I did some simple experimentation and found that if I put the database in archivelog mode, the standby creation is just hunky dory. So again, why would the GUI not do that? Very simple to do. And why the ultra cryptic error?
Archivelog mode, check. Create Standby; finally, the GUI goes to the next screen and asks for some specifics. I want a standby on a remote host, using ASYNC LOG transport (MAXIMIZE Performance). I fill in passwords and click Submit. I am treated with a status of sorts ("View Job") and after 10 minutes or so (yes, a small database), I have a fresh, hot out of the oven standby database with a dataguard broker all set up. Cool beans!
The dataguard management screens are adequate. You can change the standby mode between Protection, Availability and Performance with a few clicks; these operations do a lot of work in the background, like create standby redo logs and reconfigure both sites. Very nice. There is an option to run a test application that generates a bit of redo (1.5 mb/sec in my case), which is another nice feature to do a really simple load test of your configuration. When in MAXIMIZE PERFORMANCE mode, I could not find a way to switch between ARCH transport and REDO; again, another easy operation that I would expect to see as part of the management screen.
Overall, I am pleased with this version of the dataguard GUI. I still have an itch over those errors, but at least the major objective was accomplished with no bloodshed. I plan to play around a little more with Failover and Switchover.
For starters, I tried to create a database. Strangely, EMGC which can clone Oracle Homes, deep-drill into performance problems, manage tablespaces, backups, workloads, stats, coffee cups, user objects, and can slice-n-dice the universe, this EMGC... I could not find a button to create a database. Maybe I missed it. Which is easy to do in EMGC, because it has so many features (most of which work quite well), so many links and buttons and gizmos and ways to get to where you think you want to go. But I could not find "Create Database". So I opted to clone instead. I was not disappointed, either. The prompts and progression of screens was very intelligent, logical and reassuring. I have found that to be the case with a number of "advanced features" in EMGC; of course, there are times when you come to expect such great performance and useability and you run head-first into a brick wall of errors. But I'll get to that in a second...
Database, check. Navigate to the Data Guard section under "Management", and the feeling that you are being well cared for continues. Create Standby. Oops, no spfile. EMGC reports that you need to create an spfile. Strange that the GUI does not offer to do that for you. It is really simple; "create spfile from pfile;" and reboot the database. Ok, spfile, check. Create Standby.
What does that mean? No details, no message, no mention of a log file. This is my most frequent and biggest complaint with EMGC to date. They often forget to inform the user where they can look for more details. "Go to the log file" you would think. This is EMGC, which of the many logfiles do you want to look in? I do some looking around and spot emoms.log, which has a couple more characters, but still quite useless:
2007-05-08 15:08:15,273 [EMUI_15_08_15_/console/database/dataguard/create] ERROR em.dataguard validate.1125 -
CreateBean: ClassNotFoundException: null
2007-05-08 15:08:15,286 [149::EMUI_15_08_15_/console/database/dataguard/create] ERROR jobs.dbclone executeQuer
y.173 - TemplateUtil.executeQuery(): SQLException: Closed Connection SQL is
SELECT status FROM v$instance
So, gritting my teeth, I file an SR. I should have a separate blog entry on that. Filing SRs for OMS is not that fun. But... let us not digress anymore.
So after 20 hours, my SR is still unsolved. Fortunately, I did some simple experimentation and found that if I put the database in archivelog mode, the standby creation is just hunky dory. So again, why would the GUI not do that? Very simple to do. And why the ultra cryptic error?
Archivelog mode, check. Create Standby; finally, the GUI goes to the next screen and asks for some specifics. I want a standby on a remote host, using ASYNC LOG transport (MAXIMIZE Performance). I fill in passwords and click Submit. I am treated with a status of sorts ("View Job") and after 10 minutes or so (yes, a small database), I have a fresh, hot out of the oven standby database with a dataguard broker all set up. Cool beans!
The dataguard management screens are adequate. You can change the standby mode between Protection, Availability and Performance with a few clicks; these operations do a lot of work in the background, like create standby redo logs and reconfigure both sites. Very nice. There is an option to run a test application that generates a bit of redo (1.5 mb/sec in my case), which is another nice feature to do a really simple load test of your configuration. When in MAXIMIZE PERFORMANCE mode, I could not find a way to switch between ARCH transport and REDO; again, another easy operation that I would expect to see as part of the management screen.
Overall, I am pleased with this version of the dataguard GUI. I still have an itch over those errors, but at least the major objective was accomplished with no bloodshed. I plan to play around a little more with Failover and Switchover.
Subscribe to:
Posts (Atom)
