Given that these are (regular) dice with six sides, for the sum of the three highest numbers to be 18, there must be at least three 6’s among the 5 rolls of the dice.
If we generalize, we could calculate probability that there are n 6’s among the n dice. The sum of these probabilities would be the probability of the sum being 18.
An easier way to solve this is to 1) calculate the probability that the sum is not 18 and 2) subtract this probability from 1.
To get a sum that is not 18, there must be 0, 1 or 2 6’s among the n dice. We calculate the probability of each occurrence:
A random variable $X$ following binomial distribution with parameter n and p its probability of 'r' succeses $\rightarrow$ $p(X=r)=C^{n}_{r} p^{r} q^{n-r}$, where p is probability of success and $q=1-p$ and $r=0,1,\dots,n$
Let $X$ is the no. of $6s$ appearing
$p=P($getting $6)=\large\frac{1}{6}$
$q=P($not getting $6)=1-p=\large\frac{5}{6}$
$P ($zero $6s) =P(X=0)=^nC_0.( \large\frac{1}{6})^0.(\frac{5}{6})^n$$=(\large\frac{5}{6})^n$
$P ($one $ 6) =P(X=1)= ^nC_1.(\large\frac{1}{6})^1. (\large\frac{5}{6})^{n-1}$$=n.\large\frac{5^{n-1}}{6^n}$
$P ($ two $6s) =P(X=2)=^nC_2.(\large\frac{1}{6})^2.( \large\frac{5}{6})^{n-2}$
Therefore, the probability of rolling a sum $= 18\:is\: 1 - [P (zero\: 6s) + P (one\: 6) + P (two \:6s)]$
$\Rightarrow\:P (sum\: = 18) = 1 - \large[\frac{5^n}{6^n}$$+n. \large\frac{5^{n-1}}{6^n}$$+ ^nC_2. \large\frac{5^{n-2}}{6^n}]$
For $n = 5$, this equals,
$P (sum = 18) = 1 - \large[\frac{5^5}{6^5}$$+5 \large\frac{5^{5-1}}{6^5}$$+^5C_2. \large\frac{5^{5-2}}{6^5}]$
$P (sum = 18, when n = 5) = 1 - \large (\frac{5}{6})^5$$\; \large ($$1 + 1+\large\frac{2}{5})$
$P (sum = 18 when\: n = 5) = \large\frac{23}{648}$