Hi Marek,
i just spotted this thread after you have replied yesterday. It seems like there are mixed up a lot of things together without considering the "right metrics" and requirements.
> in top 20 selects sorted by DB time … but it is to show that system is being tuned
Oracle DB Time <> DB Response Time (from client perspective). So the described approach is basically used to reduce the CPU or I/O load on the database server and not for "performance tuning" (e.g. PX can produce much more load, but may be faster in consequence). I have written a blog post about this topic and its misconception here: [Oracle / SAP] Thinking clearly about performance and why ASH / AWR based analysis may not be the right approach
> 17 000 user calls per second and daily around 2 million dialog steps + 1,5 million of RFC+UPD+BGD steps together.
You mentioned I/O queues and possible bottlenecks, but the mentioned key metrics have no relation to it (e.g. one database FETCH call can issue 50.000 I/Os or no I/O at all or think about DB backups, etc.). So you can not correlate any user calls per second to a specific CPU or I/O load.
> During high workload load average is between 40 and 50 … Regarding CPU - current 12 CPU cores are still enough - but we are going to replace hosts due to growing business and as during problematic situation system has been like a snow ball with waiting for I/O and normal CPU processing. Except problems due to bugs (SQL code, storage firmware bug) we haven't performance issues. We expect to have load twice higher during next year - thats the reason why I'm aware of performance.
So you have some specific issue like "a snow ball with waiting for I/O and normal CPU processing". Such issues could be profiled perfectly and drilled down with a tool called "perf" in newer kernel releases. I would try to figure out the root cause first, before switching anything just based on assumptions (as far as i get it right).
> Second thing with architecture is that I would avoid going to the RAC as system is not so big yet
Absolutely - if you can handle the current workload with 12 CPUs there is absolutely no reason to switch to RAC (for scale out reasons). Todays x86 commodity hardware can scale up to much more than your current hardware configuration. You have to generate a tremendous amount of load for the need of scale out solutions like RAC. That's it from a load perspective - however there maybe other reasons for switching to RAC depending on the business requirements (OS upgrade with no downtime, rolling upgrades, consolidation, etc.). It is also very easy to degrade performance with RAC, if you do not look carefully (e.g. cache fusion with 3-way block gets in worst case, etc.).
> Does anyone know any bigger SAP installations on ASM (w/o RAC or with RAC) on Linux (not on Exadata) - if yes please - post pros and cons.
Any bigger than what? You have provided no IOPs values or anything like that. Yes, i know / have seen a lot of "large" mission critical systems on ASM. "Pros" and "Cons" depend on the infrastructure, the used features and the internal knowledge (of each team member).
> Does anyone used ASM with SLES HA ?
Yes, you can use SLES HA (Pacemaker) with Oracle Restart (as ASM is part of the GI stack nowadays) and do the ASM disk group handling by HA script (as it needs to be mounted exclusively by non clustered ASM). In consequence each database need its own ASM disk group(s) and you need to test it very carefully.
> As I have other systems with around 1 milion dialog steps per day working on vSphere 5 without issues. Please let me know if you have experience with ERP systems 2 milions (and average more than 15 000 user calls in Oracle per second) or more dialog steps per day on VMWare.
You are looking at the wrong key metrics again. The amount of dialog steps has no correlation to the important factors like CPU or I/O load. You need to measure the (OS based) load of your current system and based on that data you decide if it is possible. VMware has limitations and some of them are increasing (existing) performance issues (e.g. vCPUs and its scheduling in case of allocation). I have built up high performance Oracle HA infrastructures on VMware (e.g. with RDM, etc.) and it runs well, but only if you are not hitting some specific limits / constellations.
I highly recommend that you collect the required data first and make your decisions based on that (and not on things like database calls or dialog steps):
- Gather necessary raw data with nmon
- Create report (from raw data) with nmon analyzer and interpret it
- Get SAPS specs of your current hardware
- Calculate grows and needed resources depending on requirements and adjust the "rule of thumb" formula (see below) with your measured values. You are pretty lucky as you can measure your "real world values" and so you can adjust the SAPs to I/O ratio easily based on your system load and scale it.
All of the following "rule of thumb" values are based on unicode systems:
- 2.5 SAPS (DB + SAP App Server) generate 1 I/O operation per second
- 1 concurrent SAP NW user generates ~9 I/Os per second
- 0.3 DB-Server (DB-centric) SAPS generate 1 I/O operation per second
- Variation of DB-SAPS : App-SAPS is significant for different SAP modules e.g., 1:3 for ERP = OK versus 1:15 for CRM = too high I/O load for DB-Server
… or …
- 0.4 I/O per second / SAPS (ERP, PI)
- 0.6 I/O per second / SAPS (BW)
- 0.2 I/O per second / SAPS (assumption for SAP Components with low I/O requirements, e.g. EP, CRM, SolMan)
- 1 concurrent active ERP user= 10 SAPS
- 1 concurrent active CRM, BW user = 20 SAPS
- Complex application mix (incl. EP, PI, BW, ERP)= 22 SAPS
Ok that are my 2 cents so far for free 
… i hope this helps to go in the right direction.
Regards
Stefan