Ministers have been briefed on major curbs to the jury service which could see juries only used to decide on cases of murder, rape or manslaughter, Sky News understands.
The move would see the majority of cases heard by a judge alone, except for the most serious or those in the "public interest".
The change is expected to be formally announced by the government next week.
Sky News previously reported the Ministry of Justice was set to remove the right to trial by jury for thousands of cases, in a controversial measure which would mark a move away from a core pillar of the criminal justice system in England and Wales.
A memo sent by David Lammy to fellow ministers, seen by the Times, says there is "no right" to jury trials in the UK and that drastic action was needed to cut the backlog of cases in the crown courts, nearing 80,000.
A spokesman for the government confirmed changes are coming soon to deal with the backlog.
Speaking to Sky News' Politics Hub programme, work and pensions secretary Pat McFadden did not deny the changes were on the way.
He said he "wouldn't be surprised if my colleagues at the [Ministry of Justice] are looking at perhaps some radical ways they could reduce that backlog".
McFadden said that backlogs in the courts system are getting too long and not showing any sign of shortening.
He referenced the saying that "justice delayed is justice denied" - implying that a non-jury trial is better than no trial at all.
In the summer, retired Court of Appeal judge, Sir Brian Leveson, made recommendations to government, including a suggestion to end jury trials for many serious offences, saying they could be dealt with by a judge alone or a judge with two magistrates.
This would have been done by creating a new intermediate court, called the Crown Court Bench Division, sitting between magistrates courts and crown courts.
The plans in the leaked document go further than that, with suggestions that offences likely to receive a sentence of up to five years would be heard by judges alone.
When Mr Lammy took up his position in September, it is understood he saw the courts as his 'crisis' to tackle, with his predecessor, Shabana Mahmood, having been seen to have gripped the immediacy of the prison overcrowding crisis.
Sir Brian suggested a judge and two magistrates should hear cases with a maximum sentence of three years.
These measures will be highly controversial, with the leader of the Conservatives Kemi Badenoch saying the move "risks fairness, undermines public trust and erodes the very foundation of our justice system".
The Criminal Bar Association said the plans would "eviscerate the jury trial as we know it".
Read more from Sky News:
Man arrested in connection to waste dump
Heathrow's £33bn third runway plan chosen
A Ministry of Justice spokesperson said: "No final decision has been taken by government.
"We have been clear there is a crisis in the courts, causing pain and anguish to victims - with 78,000 cases in the backlog and rising - which will require bold action to put right."
(c) Sky News 2025: Ministers briefed on major curbs to jury service, Sky News understands

Zelenskyy is racing to beat Donald Trump's peace plan deadline – but what will Russia do?
Budget 2025: Rachel Reeves vows to 'take fair and necessary choices' and 'action on cost of living'
Army pauses use of Ajax armoured fighting vehicles after dozens of soldiers fall ill
Man arrested in connection with massive illegal waste dump in Kidlington, Oxfordshire